[SOLVED] read-only file system problem

CollieJim · 10-16-2016, 11:12 PM

I have a small computer that records data from my solar power system to a Compact Flash card. Recently a problem with one of the temporary files resulted in a new file created every minute. There are now over 148,000 such files that were never consolidated into daily files. A sample of trying to remove 2 of them follows.

I need to delete these files and try to bypass (rename?) the bad file. For this I have removed the card and mounted in on my desktop computer which has LinuxMint 17.3 on it. However when I try to delete, I get a read-only file system error.

Code:

/mnt # mount -o rw /dev/sdb3 tmp
/mnt # cat /etc/mtab | grep sdb3 -
/dev/sdb3 /mnt/tmp ext2 rw 0 0
/mnt # ls -al
total 24
drwxr-xr-x  6 root root 4096 Mar 12  2016 .
drwxr-xr-x 25 root root 4096 Jul 16 16:27 ..
drwxr-xr-x  8 root root 4096 Oct  1 19:22 Backups
drwxrwsr-x  5 1001 1001 4096 Nov  8  2015 tmp
/mnt # cd tmp
/mnt/tmp # ls -al
total 6220
drwxrwsr-x 5 1001 1001    4096 Nov  8  2015 .
drwxr-xr-x 6 root root    4096 Mar 12  2016 ..
drwx------ 2 root root  294912 Mar 29  2012 lost+found
drwxr-xr-x 5 root root 6045696 Oct 17 10:57 mySystem
drwxr-sr-x 3 root 1001    4096 Nov  8  2015 to-be-deleted-maybe
/mnt/tmp # cd mySystem/
/mnt/tmp/mySystem # ls -al gmStatus11111*
-rw-r--r-- 1 root root 42741 Jun 23 20:17 gmStatus11111202627995826gms
-rw-r--r-- 1 root root 63016 Aug  5 16:23 gmStatus1111193369457273811gms
/mnt/tmp/mySystem # rm -rf gmStatus11111*
rm: cannot remove ‘gmStatus11111202627995826gms’: Read-only file system
rm: cannot remove ‘gmStatus1111193369457273811gms’: Read-only file system
907 11:50:53 /mnt/tmp/mySystem #

I see that the owner:group of /mnt/tmp is 1001.1001. Before the mount it was 0.0. Since root mounted the file system, why is this so? Is it related to my problem?

How do I proceed?

Thanks
Jim

rtmistler · 10-17-2016, 08:33 AM

Those are not directories, they are files, you do not need the -r directive.

All the owners which are 1001, you can use the chown command to change the ownership back to root and there is a -R directive for recursing.

The question is whether or not this occurs regularly or if this is just a one-of occurrence which happened.

rknichols · 10-17-2016, 08:51 AM

"Read-only filesystem" is not a permissions issue. The mount has become read-only, either because of errors in the filesystem or problems in the device itself. If you run "grep sdb3 /proc/mounts" you should see it as "ro". There may be some clue as to why in the messages in /var/log/syslog.

CollieJim · 10-17-2016, 09:30 AM

Thanks for the replies.

Disk errors were the problem. I ran fsck, which found and fixed many errors.

The files are now being deleted, but at anywhere from 10 files/second to 6 seconds per file it's going to take a while to get rid of 148,000 files. It would be really nice if there were a utility that could edit the directory and do it all at once...

rtmistler · 10-17-2016, 10:00 AM

Quote:

Originally Posted by CollieJim

Thanks for the replies.

Disk errors were the problem. I ran fsck, which found and fixed many errors.

The files are now being deleted, but at anywhere from 10 files/second to 6 seconds per file it's going to take a while to get rid of 148,000 files. It would be really nice if there were a utility that could edit the directory and do it all at once...

Given that this is a small computer, perhaps a non-MMU system if it is an ARM processor, and that you may have little free space on disk, it may be that these are the problems for file deletion times being lengthy, and once you free up more disk space you'll find that the deletions are faster. Unsure because I do not believe you cited whether or not disk space was near the limits versus not.

CollieJim · 10-17-2016, 10:29 AM

The computer used for the deletions was an HP laptop, and the files were on a 16GB Compact Flash card. The 12GB partition was just over half full. The computer that creates the files has an Intel processor and uses Voyage Linux (http://linux.voyage.hk).

rknichols · 10-17-2016, 10:33 AM

Sometimes the fastest way to delete files is to save the ones you want to keep and then mkfs a new filesystem.

CollieJim · 10-17-2016, 10:46 AM

I've done that before, but this time there were too many files to save.

The card has been used continuously for over 5 years, with new data stored at least once a minute. I think it may be time to use a new card.

rtmistler · 10-17-2016, 12:23 PM

Quote:

Originally Posted by CollieJim

I've done that before, but this time there were too many files to save.

The card has been used continuously for over 5 years, with new data stored at least once a minute. I think it may be time to use a new card.

Yes, NAND storage like that will have a lifetime on the individual storage cells and eventually they'll wear down. Those types of disks are not S.M.A.R.T. enabled, thus you have no real way to tell their "current" capacity of the propensity of the disk to fail soon versus not so soon.

Gamesmedic · 06-20-2018, 02:32 PM

I bought a 3 tbyte hard drive and tried to clone my mint system not realizing thee trouble I was getting into.
The clone job failed, and I had no idea why. The error came up damaged disk sectors.
I reaLized later that most linux distros and msdos systems havent overcome the 2 terabyte limit.
Even Gparted can't seem to.

My original system now only boots into terminal. I read up on this error by trying sstartx which only resulted in another error of unable to lock file /tmp/.x-ot.
Upon further checking *grep sda1 /proc/mounts the read out is
/dev/sda1 / et4 ro,relatime,data=ordered 0 0
The system is now read only. Clownzilla didn't finish unlocking it after the failure.
How do I put the system correct again and make it rw?

Gamesmedic · 06-20-2018, 02:50 PM

Found the solution. Grub was instructing vmlinux (kernal) to go into read only mode.
It was pathetically simple to edit grub menu from ro to rw... The smallest things... ARGGG

rknichols · 06-20-2018, 07:23 PM

Quote:

Originally Posted by Gamesmedic

I bought a 3 tbyte hard drive and tried to clone my mint system not realizing thee trouble I was getting into.
The clone job failed, and I had no idea why. The error came up damaged disk sectors.
I reaLized later that most linux distros and msdos systems havent overcome the 2 terabyte limit.
Even Gparted can't seem to.

The 2 TiB limit for disks partitioned with a legacy MBR partition table is not something that Linux or any other OS can overcome. The fields for LBA addresses in that partition table are 32 bits, and for disks that use 512-byte logical sector size (the majority of disks today), that works out to a 2 TiB limit**. For a 3 TB disk, you need to use GPT partitioning.

**Actually, the strict limitation is that no partition can start beyond the 2 TiB point. In concept, you could have a partition that starts just below 2 TiB, at LBA 0xFFFFFFFF, and extends for another 0xFFFFFFFF sectors giving you just under 4 TiB total, but you may have trouble finding a partitioning tool that will let you do that. Also, an MBR partition table on a disk that uses a 4K logical sector size (4K-native or 4Kn format) can support up to 16 TiB.

Gamesmedic · 06-21-2018, 09:54 AM

Thank you for your advice. I tried splitting the partition into two. What I ran into is the partition manager (Gparted) failing because the ending LBA on the second partition was higher than the maximum LBA of 4294967299 and would not allow ANY partition higher than that. So essentially, it would be a 3 tbyte disk with only 2 being usable. I went ahead and partitioned it to GPT to get the full use of it, but its not usable as a bootable disk. I'm keeping my system on the 320GB disk and just using the 3TB disk for data/media/documents storage and backup system images.

Now, onto my last problem. Grub still boots into the OS in read only (thank you clownzilla). Could anyone tell me how to edit this new Grub2? Such a pain in the *ss! At least with Grub 1.0 I could use Gedit. Running Mint Silvia (Cinnamon)

rknichols · 06-21-2018, 02:01 PM

You certainly can boot from a from a GPT disk. If you are doing manual partitioning and your system is booting in legacy BIOS (non-UEFI) mode, you do have to create a small bios boot partition where GRUB can be embedded. There's some help at https://forums.linuxmint.com/viewtopic.php?t=242282 plus other useful links returned by a search for linux+mint+gpt.

GRUB is intentionally configured to tell the kernel to mount the root filesystem initially in read-only mode. If all goes well, the code in the initrd should switch that to read/write. It's only when something else goes wrong during the boot sequence that the filesystem is left read-only. Are there any messages displayed that give a hint about what is wrong?

Gamesmedic · 06-21-2018, 04:34 PM

What caused this mess was Clownzilla crashing during a cloning process and I manually had to reboot.

A little background on my "solution";

1. I edit the Grub during boot, on the fly, to open the OS in read write mode by changing a string in a line from "ro" to "rw".

Failing to do so will boot me to the command prompt and

Code:

startx

fails to get a lock on /tmp/.x-ot (since its in read only mode). What clued me in the first place was logging in at the prompt and doing a

Code:

sudo apt-get update

and getting nothing but failed (Read only file system) errors.

Doing a

Code:

sudo mount -o / remount,rw (I think this was the code I used)

at the prompt brings up the GUI login screen, but after "logging in" I don't get to the desktop. The screen goes black for a second then comes right back to the GUI login screen because it doesn't seem to let me login, but the username and password are correct.

2. When I do accomplish booting to the desktop everything runs as usual.

This is fine, except when I Hibernate, Shutdown, or restart. After hibernate the system boots up and for some reason gets stuck on the sending ata signal, no reply, after 3 times it "giving up", and just sits there until I turn the machine off.

Restarting simply sends me to a black screen until I again, shut the machine off.

Shutting down is the only thing that is working, but then I have to edit GRUB on the fly again to get back to the desktop. Hypothetically speaking, this could be a great way of keeping unwanted snoops from getting into my computer, but since that really isn't an issue for me, these extra steps are making things more difficult than they ought to.

BTW, doing a

Code:

sudo apt-get update

in terminal has no effect.
Thank you again for the reply. I will looking that thread. I would really like to use this HD to boot from. I'm even thinking of getting a second drive and running a RAID system so I don't have to deal with this BS of failed drives anymore. On the flip side, I've always wanted to ditch Windoze in the past. This latest drive failure finally pushed me to it.