Trying to stand up a local mirror for my company, issues with rsync
I am roughly basing what I am trying to do around this article: https://blog.programster.org/centos-...entos-7-mirror
tldr: Step 1. rsync with a CentOs mirror and a EPEL mirror Step 2. stand up company internal CentOs mirror with nginx Step 3. Profit I have been going through and trying to pick mirrors near me to rsync with, but it seems all of them at one point or another hang on a file and die. I checked with the networking team, the corporate firewall isn't known to be blocking my rsync, and I am able to get a lot of files before it chokes, so that's not it. Sometimes switching mirrors fixes the problem, sometimes it doesn't. Here is an example of the command I am running to sync EPEL: sudo rsync -avSHPz --delete rsync://mirror.team-cymru.com/epel/7/x86_64 ./CentOSEpelMirror/7/x86_64 which yields the following error: x86_64/Packages/o/openarena-0.8.8-7.el7.noarch.rpm 335,753,632 83% 26.23MB/s 0:00:02 rsync: connection unexpectedly closed (3910004 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [receiver=3.1.2] rsync: connection unexpectedly closed (522056 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(235) [generator=3.1.2] The only thing I can think of is that it's an issue with WSL, since that is how I am rsyncing, but I was wondering if anyone had any additional input for me. I was about to write a python script to parse the mirror list and get me all the rsync urls, and then just write a bash script that loops over all of them whenever rsync chokes and dies. |
The syntax looks wrong. Try it with a dry run first before deleting anything. Mind the slashes, too.
Code:
rsync -avSHPz --dry-run user@mirror.team-cymru.com:/epel/7/x86_64/ ./CentOSEpelMirror/7/x86_64/ Substitute "user" for the actual remote account. I see that SSH keys appear to not be used there. They would be a good idea. However, keep in mind that if things were set up properly that remote root access was disabled. By the way, you write that you are using CentOS. What does the Linux Subsystem for Windows (WSL) have to do with any of that? |
Quote:
|
WSL is a weird distro and as far as I know is using paravirtualization, to add to the strangeness. So a lot of things might not work, which I expect is a part of their plan.
Back to rsync, the following should work if WSL is not too broken: Code:
test -e ./CentOSEpelMirror/7/x86_64/ || mkdir -p ./CentOSEpelMirror/7/x86_64/ If that does not work, then try on a normal distro like CentOS, Debian, Devuan, etc on bare metal. |
While it seems you can run a webserver in WSL I think it would be a lot easier to create a local repository with a real distribution using virtualization software like VirtualBox.
I have not played with WSL but without knowing where or how it uses . as suggested use an absolute path. Or install a web server i.e. IIS and create a local repository on Windows itself. |
Quote:
|
Quote:
|
Ok. It seems then that you're stuck in a shop with resellers in place of support. Just keep in mind that WSL is not intended to give an honest demonstration of what a real distro could do. It appears to work barely enough to give Windows users a bad experience and have them turn back to the familiar, eschewing the modern for another work generation. Thus the abysmally low adoption rate for WSL. So, you might be able to get it to Rsync, you might not. Keep us posted.
A mirror of a repository is not important data, in that it it can be easily replaced if lost and it is cryptographically signed so does not need to be locked away. So if you have a hardware budget even a simple Raspberry Pi with a USB hard drive would do. Or an old desktop computer with a new or newish hard drive would do just as well. However, I'm guessing that this repository is to be mirrored on an existing web server? |
Quote:
So at least now I have the evidence to show my manager and security that the issue is definitely with winblows, and I am sure they will be able to stand up a vm somewhere to do this. Their issue is not with the security of the mirror itself, but more of if you let your users do whatever they want on your corporate network, then if someone malicious comes in it is much easier for them to attack your network. Anyways thanks for the help you all. |
All times are GMT -5. The time now is 06:50 AM. |