Backup fails: "A nonrecoverable I/O error occurred on file"

We have been having a problem with one of our backups:
A nonrecoverable I/O error occurred on file "D:\Program Files\Microsoft SQL
Server\MSSQL.1\MSSQL\Data\SOMEDB_Data.mdf"
It's on the same external storage drive as all of our databases yet it's the
only one that errors out, so it appears that the problem is with the .mdf
file, not with the hard disk. And it only errors intermittently...sometimes
the backup succeeds.
We have:
Run dbcc checkdb and receive no errors.
Dropped the database and restored from backup thinking the .mdf file would
be recreated.
We're not sure what to try next. Again, it doesn't seem like a hardware
problem but I can't rule it out either. Please advise."nexdeveloper" <nexdeveloper@.community.nospam> wrote in message
news:E0948BFC-C002-4EBF-9E00-B0EAF6A8E850@.microsoft.com...
> We have been having a problem with one of our backups:
> A nonrecoverable I/O error occurred on file "D:\Program Files\Microsoft
> SQL
> Server\MSSQL.1\MSSQL\Data\SOMEDB_Data.mdf"
> It's on the same external storage drive as all of our databases yet it's
> the
> only one that errors out, so it appears that the problem is with the .mdf
> file, not with the hard disk. And it only errors
> intermittently...sometimes
> the backup succeeds.
> We have:
> Run dbcc checkdb and receive no errors.
> Dropped the database and restored from backup thinking the .mdf file would
> be recreated.
> We're not sure what to try next. Again, it doesn't seem like a hardware
> problem but I can't rule it out either. Please advise.
Is this a SAN drive per chance? Or any special SCSI drive that may be doing
something while you do a backup?
Otherwise you may need to open a ticket with MS.
Greg Moore
SQL Server DBA Consulting Remote and Onsite available!
Email: sql (at) greenms.com http://www.greenms.com/sqlserver.html|||Hello,
Since the issue only occurs iwht one mdf file, it might be a corruption
issue. Since the issue also occurs with a restore from backup, you may try
the following steps to troubleshoot the issue:
1. Create a new, blank database with whatever name and filename you want
(we may rename it later)
2. Generate the scripts of all tables and create them in the new database;
3. Transfer all data into the new database;
4. Check if the issue occurs with the new database
If the issue does not occur any more, you may want to detach the old
database, and rename the new database to the original name to test.
If the issue persists, you may want to use hardware check tool to confirm
there is no hardware related issue.
To find out the root cause of this issue we may need to analyze memory
dumps, this work has to be done by contacting Microsoft Product Support
Services. Therefore, we probably will not be able to resolve the issue
through the newsgroups. I recommend that you open a Support incident with
Microsoft Product Support Services so that a dedicated Support Professional
can assist with this case. If you need any help in this regard, please let
me know.
For a complete list of Microsoft Product Support Services phone numbers,
please go to the following address on the World Wide Web:
http://support.microsoft.com/directory/overview.asp
If you have any concerns or comments on this, please feel free to let's
know. Thank you.
Best Regards,
Peter Yang
MCSE2000/2003, MCSA, MCDBA
Microsoft Online Community Support
==================================================Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications
<http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx>.
Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
<http://msdn.microsoft.com/subscriptions/support/default.aspx>.
==================================================This posting is provided "AS IS" with no warranties, and confers no rights.|||Yes this a SAN drive at a remote datacenter. The backup runs in the middle
of the night so I don't think the SAN has any heavy lifting going on at the
time the backup runs. We're going to continue to investigate and open a
ticket with MS if appropriate.
"Greg D. Moore (Strider)" wrote:
>
>
> "nexdeveloper" <nexdeveloper@.community.nospam> wrote in message
> news:E0948BFC-C002-4EBF-9E00-B0EAF6A8E850@.microsoft.com...
> > We have been having a problem with one of our backups:
> >
> > A nonrecoverable I/O error occurred on file "D:\Program Files\Microsoft
> > SQL
> > Server\MSSQL.1\MSSQL\Data\SOMEDB_Data.mdf"
> >
> > It's on the same external storage drive as all of our databases yet it's
> > the
> > only one that errors out, so it appears that the problem is with the .mdf
> > file, not with the hard disk. And it only errors
> > intermittently...sometimes
> > the backup succeeds.
> >
> > We have:
> >
> > Run dbcc checkdb and receive no errors.
> > Dropped the database and restored from backup thinking the .mdf file would
> > be recreated.
> >
> > We're not sure what to try next. Again, it doesn't seem like a hardware
> > problem but I can't rule it out either. Please advise.
> Is this a SAN drive per chance? Or any special SCSI drive that may be doing
> something while you do a backup?
> Otherwise you may need to open a ticket with MS.
>
> --
> Greg Moore
> SQL Server DBA Consulting Remote and Onsite available!
> Email: sql (at) greenms.com http://www.greenms.com/sqlserver.html
>
>|||We're going to continue to investigate and open a ticket with MS if
appropriate. Thanks for the response...I'll let you know if I have further
questions.
"Peter Yang [MSFT]" wrote:
> Hello,
> Since the issue only occurs iwht one mdf file, it might be a corruption
> issue. Since the issue also occurs with a restore from backup, you may try
> the following steps to troubleshoot the issue:
> 1. Create a new, blank database with whatever name and filename you want
> (we may rename it later)
> 2. Generate the scripts of all tables and create them in the new database;
> 3. Transfer all data into the new database;
> 4. Check if the issue occurs with the new database
> If the issue does not occur any more, you may want to detach the old
> database, and rename the new database to the original name to test.
> If the issue persists, you may want to use hardware check tool to confirm
> there is no hardware related issue.
> To find out the root cause of this issue we may need to analyze memory
> dumps, this work has to be done by contacting Microsoft Product Support
> Services. Therefore, we probably will not be able to resolve the issue
> through the newsgroups. I recommend that you open a Support incident with
> Microsoft Product Support Services so that a dedicated Support Professional
> can assist with this case. If you need any help in this regard, please let
> me know.
> For a complete list of Microsoft Product Support Services phone numbers,
> please go to the following address on the World Wide Web:
> http://support.microsoft.com/directory/overview.asp
> If you have any concerns or comments on this, please feel free to let's
> know. Thank you.
> Best Regards,
> Peter Yang
> MCSE2000/2003, MCSA, MCDBA
> Microsoft Online Community Support
> ==================================================> Get notification to my posts through email? Please refer to
> http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
> ications
> <http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx>.
> Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
> where an initial response from the community or a Microsoft Support
> Engineer within 1 business day is acceptable. Please note that each follow
> up response may take approximately 2 business days as the support
> professional working with you may need further investigation to reach the
> most efficient resolution. The offering is not appropriate for situations
> that require urgent, real-time or phone-based interactions or complex
> project analysis and dump analysis issues. Issues of this nature are best
> handled working with a dedicated Microsoft Support Engineer by contacting
> Microsoft Customer Support Services (CSS) at
> <http://msdn.microsoft.com/subscriptions/support/default.aspx>.
> ==================================================> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>|||"nexdeveloper" <nexdeveloper@.community.nospam> wrote in message
news:B86FFC11-4C68-462B-88B5-3C9B640369DF@.microsoft.com...
> Yes this a SAN drive at a remote datacenter. The backup runs in the
> middle
> of the night so I don't think the SAN has any heavy lifting going on at
> the
> time the backup runs. We're going to continue to investigate and open a
> ticket with MS if appropriate.
Was thinking more perhaps the SAN drive might be doing something like
breaking a mirror during that time to do a backup and then recreating the
mirror or some other function and not properly quiescing the database.
> "Greg D. Moore (Strider)" wrote:
>>
>>
>> "nexdeveloper" <nexdeveloper@.community.nospam> wrote in message
>> news:E0948BFC-C002-4EBF-9E00-B0EAF6A8E850@.microsoft.com...
>> > We have been having a problem with one of our backups:
>> >
>> > A nonrecoverable I/O error occurred on file "D:\Program Files\Microsoft
>> > SQL
>> > Server\MSSQL.1\MSSQL\Data\SOMEDB_Data.mdf"
>> >
>> > It's on the same external storage drive as all of our databases yet
>> > it's
>> > the
>> > only one that errors out, so it appears that the problem is with the
>> > .mdf
>> > file, not with the hard disk. And it only errors
>> > intermittently...sometimes
>> > the backup succeeds.
>> >
>> > We have:
>> >
>> > Run dbcc checkdb and receive no errors.
>> > Dropped the database and restored from backup thinking the .mdf file
>> > would
>> > be recreated.
>> >
>> > We're not sure what to try next. Again, it doesn't seem like a
>> > hardware
>> > problem but I can't rule it out either. Please advise.
>> Is this a SAN drive per chance? Or any special SCSI drive that may be
>> doing
>> something while you do a backup?
>> Otherwise you may need to open a ticket with MS.
>>
>> --
>> Greg Moore
>> SQL Server DBA Consulting Remote and Onsite available!
>> Email: sql (at) greenms.com
>> http://www.greenms.com/sqlserver.html
>>
Greg Moore
SQL Server DBA Consulting Remote and Onsite available!
Email: sql (at) greenms.com http://www.greenms.com/sqlserver.htmlsql