|
»
ZFS in productions 64 bit
|
View:
New views
14 Messages
—
Rating Filter:
Alert me
|
 |
ZFS in productions 64 bit
by tonix (Antonio Nati)
::
Rate this Message:
Is anyone using in heavy production environment a ZFS FS with AMD 64 bit?
Thanks, Tonino -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@... ------------------------------------------------------------ _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: ZFS in productions 64 bit
by Freddie Cash-8
::
Rate this Message:
On Mon, Jul 6, 2009 at 7:27 AM, Tonix (Antonio Nati)
<tonix@...>wrote: > Is anyone using in heavy production environment a ZFS FS with AMD 64 bit? > We're using FreeBSD 7.2 on our backup servers. The primary backup server does remote backups for over 105 servers, every night. And then pushes the changes to the secondary backup server, every day. Both servers are: 5U Chenbro case, with 24 hot-swappable SATA drive bays 1350 watt, 4-way redundant PSU (yes, it's overkill) Tyan h2000M motherboard 2x AMD Opteron 2220 CPUs @ 2.8 GHz (dual-core) 3Ware 9550SXU-12ML PCI-X RAID controller 3Ware 9650SE-12ML PCIe RAID controller Intel PRO/1000MT PCI-X quad-port gigabit NIC 24x 500 GB SATA harddrives 2x CompactFlash drives in CF-to-IDE or CF-to-SATA adapters The CompactFlash are configured using gmirror, and hold / and /usr. (/usr is there as we originally had some issues booting into single-user mode and getting the zpool up and running.) zpool is configured with 3x raidz2 vdevs, each vdev uses 8 harddrives. Gives us ~ 10 TB usable space in the pool. Everything other than / and /usr are ZFS, including /usr/src, /usr/obj, /usr/ports, /var, /tmp, /usr/local, /home, and so on. Over the course of a backup run, we average 80 MBytes/sec writes, which is limited by the horrible upload performance of the remote ADSL sites. We've benchmarked the system maxing out at 550 MBytes/sec write and 5.5 GBytes/sec read. We had to do a lot of manual tuning when we started out, to limit vm.kmem_size_max and vfs.zfs.arc_max, and to disable prefetch (vfs.zfs.prefetch_disable=1), as we started with 7-STABLE shortly after 7.0 was released. With FreeBSD 7.2, we've removed the tuning, but left prefetch disabled (with prefetch enabled, we'd lockup the system after about 5 hours of heavy rsync usage ... no swap space left). Our backups are done using rsync. We serialise the backups of the systems at each remote site, but run the backups for multiple sites in parallel. The only "non-standard" change we made was to switch to openssh-portable from ports, and enable the HPN patches. We saw our rsync throughput go up by 30% after tuning the network sysctls, using HPN. Other than trying to use USB sticks instead of CompactFlash originally, during the initial tuning phase, and when experimenting with prefetch, the system has been rock solid. Our next big ZFS project will be using similar hardware to create our own SAN setup, using iSCSI exports, for a virtualisation setup (Linux+KVM on the processing nodes, FreeBSD+ZFS on the storage nodes). -- Freddie Cash fjwcash@... _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: ZFS in productions 64 bit
by Steve Bertrand-2
::
Rate this Message:
Freddie Cash wrote:
> On Mon, Jul 6, 2009 at 7:27 AM, Tonix (Antonio Nati) > <tonix@...>wrote: > >> Is anyone using in heavy production environment a ZFS FS with AMD 64 bit? [...snip...] > We've > benchmarked the system maxing out at 550 MBytes/sec write and 5.5 GBytes/sec > read. Holy crap! Would you be so kind as to share some of your relevant pieces of config on how you have the OS and ZFS set up? > Our backups are done using rsync. We serialise the backups of the systems > at each remote site, but run the backups for multiple sites in parallel. Although not nearly as impressive (nor 64-bit), I've been using ZFS on my AMANDA backup server since 7.0, and it too (after a bit of tweaking) has been rock solid. > Our next big ZFS project will be using similar hardware to create our own > SAN setup, using iSCSI exports, for a virtualisation setup (Linux+KVM on the > processing nodes, FreeBSD+ZFS on the storage nodes). Please do make notes ;) Steve |
|
|
Re: ZFS in productions 64 bit
by Freddie Cash-8
::
Rate this Message:
On Tue, Jul 7, 2009 at 5:46 AM, Steve Bertrand <steve@...> wrote:
> Freddie Cash wrote: > > On Mon, Jul 6, 2009 at 7:27 AM, Tonix (Antonio Nati) > > <tonix@...>wrote: > > > >> Is anyone using in heavy production environment a ZFS FS with AMD 64 > bit? > > [...snip...] > > > We've > > benchmarked the system maxing out at 550 MBytes/sec write and 5.5 > GBytes/sec > > read. > > Holy crap! > > Would you be so kind as to share some of your relevant pieces of config > on how you have the OS and ZFS set up? > http://forums.freebsd.org/showthread.php?t=3689 All the gory details of how we do our backups for over 100 remote servers every night, in under 5 hours, with daily snapshots going back just shy of 6 months (average size of the daily snapshots is 10 GB). If we had 10 Mbps symmetrical connections to our remote sites instead of 2.0/0.5 Mbps ADSL, we could probably drop the backup time to an hour. :) If there's anything missing from there that you would like to know, just ask. :) -- Freddie Cash fjwcash@... _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
|
|
|
Re[2]: ZFS in productions 64 bit
by Dennis Yusupoff
::
Rate this Message:
Good day, Freddie.
> If there's anything missing from there that you would like to know, just > ask. :) At first, I would like to say thanks for your detailed "success-story" report. It was great! So, now a questions. ;) Have you got any HDD failure, and if yes, how do you repair filesystem and so on? Why are you use software RAID, not hardware? -- С уважением, системный администратор Ozerki.Net/Cifracom.Ru Юсупов Денис mailto:dyr@... _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: ZFS in productions 64 bit
by tonix (Antonio Nati)
::
Rate this Message:
Freddie Cash ha scritto:
> On Tue, Jul 7, 2009 at 9:24 AM, Tonix (Antonio Nati) > <tonix@...>wrote: > > >> Freddie Cash ha scritto: >> >> On Mon, Jul 6, 2009 at 7:27 AM, Tonix (Antonio Nati)<tonix@...> <tonix@...>wrote: >> >> >> >> Is anyone using in heavy production environment a ZFS FS with AMD 64 bit? >> >> >> >> We're using FreeBSD 7.2 on our backup servers. The primary backup server >> does remote backups for over 105 servers, every night. And then pushes the >> changes to the secondary backup server, every day. >> >> >> >> >> Are you using ZFS only on backup servers, or also on remote servers to make >> a snaphost of data to be backed up? >> >> > > Only on the backup servers. The remote servers are running either Debian > Linux 4.0, FreeBSD 6.x/7.x, or RHEL 4.x. And we do a couple of manual > backups of Windows XP stations using rsync for Windows. > I'm evaluating whether to use ZFS for main NFS storage I will provide to all front-end servers. Possibility to snapshost partitions, to extend/decrease them is something I'd love to do, but I'm wondering on reliability on long term. I see main concerns are about 32bit servers, while 64bits servers looks to be more 'protected', but I'm not really sure about. Better to wait for FBSD 8.0? Thanks for any advice. Tonino -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@... ------------------------------------------------------------ _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: ZFS in productions 64 bit
by Freddie Cash-8
::
Rate this Message:
On Tue, Jul 7, 2009 at 9:49 AM, Tonix (Antonio Nati)
<tonix@...>wrote: > Freddie Cash ha scritto: > >> On Tue, Jul 7, 2009 at 9:24 AM, Tonix (Antonio Nati) >> <tonix@...>wrote: >> >>> Freddie Cash ha scritto: >>> On Mon, Jul 6, 2009 at 7:27 AM, Tonix (Antonio Nati)< >>> tonix@...> <tonix@...>wrote: >>> Is anyone using in heavy production environment a ZFS FS with AMD 64 >>> bit? >>> >>> We're using FreeBSD 7.2 on our backup servers. The primary backup >>> server >>> does remote backups for over 105 servers, every night. And then pushes >>> the >>> changes to the secondary backup server, every day. >>> >>> Are you using ZFS only on backup servers, or also on remote servers to >>> make >>> a snaphost of data to be backed up? >>> >> >> Only on the backup servers. The remote servers are running either Debian >> Linux 4.0, FreeBSD 6.x/7.x, or RHEL 4.x. And we do a couple of manual >> backups of Windows XP stations using rsync for Windows. >> >> > I'm evaluating whether to use ZFS for main NFS storage I will provide to > all front-end servers. > Possibility to snapshost partitions, to extend/decrease them is something > I'd love to do, but I'm wondering on reliability on long term. I see main > concerns are about 32bit servers, while 64bits servers looks to be more > 'protected', but I'm not really sure about. > > Better to wait for FBSD 8.0? > > Thanks for any advice. > ZFS on 64-bit FreeBSD 7.2 has been extremely stable for us. A lot of bug fixing, performance issues, and memory issues were fixed in ZFS with 7.2. The version of ZFS included in 7.2 is ZFSv6. FreeBSD 7-STABLE includes an upgrade of ZFS to ZFSv13. This will be available in FreeBSD 7.3, and is the same as what will be in FreeBSD 8.0. The main difference in ZFS support between 7.x and 8.x is that 8.x includes support for booting directly from ZFS, so you can run ZFS-only systems. We haven't stress tested NFS support, but have used it to share out a couple of directories here and there (to Linux clients), and it seems to be as stable/usable as normal NFS in FreeBSD. AFAICT, ZFS uses the normal NFS support in FreeBSD to do the sharing, and the sharenfs property for ZFS filesystems just configures the exports file for you. 64-bit installs with lots of RAM are recommended to get the most out of ZFS, although it is perfectly usable on 32-bit systems. There are reports of people using it on laptops and on 32-bit systems with as little as 768 MB of RAM. (I use it on my home media server which is 32-bit with 2 GB of RAM). On 32-bit systems, you have to do a lot of manual tuning of loader.conf tunables to limit how much memory ZFS can use. But on 64-bit systems running FreeBSD 7.2 or newer, with at least 2 GB of RAM, it can auto-tune itself (although you still need to disable prefetch). I'd recommend giving it a test run. You might be surprised. :) Or it might not work in your situation. :) -- Freddie Cash fjwcash@... _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: Re[2]: ZFS in productions 64 bit
by Freddie Cash-8
::
Rate this Message:
On Tue, Jul 7, 2009 at 9:31 AM, Dennis Yusupoff <dyr@...> wrote:
> > If there's anything missing from there that you would like to know, just > > ask. :) > > At first, I would like to say thanks for your detailed "success-story" > report. It was great! > So, now a questions. ;) > Have you got any HDD failure, and if yes, how do you repair filesystem > and so on? > We've had one drive fail so far, which is how we discovered that our intial pool setup was horribly, horribly, horribly misconfigured. We originally used a single raidz2 vdev using all 24 harddrives. NOT RECOMMENDED!!! Our throughput was horrible (taking almost 8 hours to complete a backup run of less than 80 servers). Spent over a week trying to get that new drive to resilver, but it just thrashed the drives. Then I found a bunch of articles online that describe how the raidz implementation works (limited to the IOps of a single drive), and that one should not use more than 8 or 9 drives in a raidz vdev. We built the secondary server using the 3-raidz vdev layout, and copied over as much data as we could (lost 3 months of daily backups, saved 2 months). Then we rebuilt the primary servers using the 3-raidz vdev layout, and copied the data back. Since then, we haven't had any other harddrive issues. And, we now run a "zpool scrub" every weekend to check for filesystem inconsistencies, bad checksums, bad data, and so on. So far, no issues found. > Why are you use software RAID, not hardware? > For the flexibility, and all the integrity features of ZFS. The pooled storage concept is just so much nicer/easier to work with than hardware RAID arrays, separate LUNs, separate volume managers, separate partitions, etc. Need more storage? Just add another raidz vdev to the pool. Instantly have more storage space, and performance increases as well (the pool stripes across all the vdevs by default). Don't have any more drive bays? Then just replace the drives in the raidz vdev with larger ones. All the space becomes available to the pool. And *all* the filesystems use that pool, so they all get access to the extra space (no reformatting, no repartitioning, no offline expansion required). Add in the snapshots feature, that actually works without slowing down the system (UFS) or requiring "wasted"/used space (LVM), and it's hard to use hardware RAID anymore. :) Or course, we do still use hardware RAID controllers, for the disk management and alerting features, the onboard cache, the fast buses (PCI-X/PCIe), multi-lane cabling, hot-plug support, etc; we just don't use the actual RAID features. All of our Linux servers still use hardware RAID (5 and 10), with LVM on top, and XFS on top of that. But it's just not as nice of a storage stack to work with. :) -- Freddie Cash fjwcash@... _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
just a quickie please
by Shannon Wheeler
::
Rate this Message:
I'm not currently subscribed to a sendmail list and
I'm leaving for vacation tomorrow so of course a problem will show up today. FreeBSD 6ish sendmail 13 SquirrelMail 1.4.13 If I log into squirrelmail, I can send mail no problem. If user is_practitioner logs in, when sending mail she receives: ERROR: Message not sent. Server replied: Requested action not taken: mailbox unavailable 550 5.1.1 <is_practitioner@...>... I don't know who you're looking for. ~~~~~~~~~~~~~~ Note that the user's logon name is: is_practitioner but her e-mail address is: isssp_practitioner@... (FreeBSD limited me to 16 characters for the username) note also that the reject message lists the main domain that the machine belongs to but her e-mail address is a seperate virtual domain. When setting up this user, I set her shell as 'mail-only' even though no such shell exists... most of my other users are set for 'nologin'. In fact, hardly any users use the webmail, especially not for sending. But it should work and does work for other users that I've tried. Anyone know exactly what the problem is? otherwise, just where to look... username vs alias problem? mail-only shell problem? something else? Thank you, Shannon Wheeler 780-790-0728 _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: ZFS in productions 64 bit
by tonix (Antonio Nati)
::
Rate this Message:
Freddie Cash ha scritto:
> On Tue, Jul 7, 2009 at 9:31 AM, Dennis Yusupoff <dyr@...> wrote: > > >>> If there's anything missing from there that you would like to know, just >>> ask. :) >>> >> At first, I would like to say thanks for your detailed "success-story" >> report. It was great! >> So, now a questions. ;) >> Have you got any HDD failure, and if yes, how do you repair filesystem >> and so on? >> >> > > > We've had one drive fail so far, which is how we discovered that our intial > pool setup was horribly, horribly, horribly misconfigured. We originally > used a single raidz2 vdev using all 24 harddrives. NOT RECOMMENDED!!! Our > throughput was horrible (taking almost 8 hours to complete a backup run of > less than 80 servers). Spent over a week trying to get that new drive to > resilver, but it just thrashed the drives. > > Then I found a bunch of articles online that describe how the raidz > implementation works (limited to the IOps of a single drive), and that one > should not use more than 8 or 9 drives in a raidz vdev. We built the > secondary server using the 3-raidz vdev layout, and copied over as much data > as we could (lost 3 months of daily backups, saved 2 months). Then we > rebuilt the primary servers using the 3-raidz vdev layout, and copied the > data back. > > Since then, we haven't had any other harddrive issues. > > And, we now run a "zpool scrub" every weekend to check for filesystem > inconsistencies, bad checksums, bad data, and so on. So far, no issues > found. > > > > >> Why are you use software RAID, not hardware? >> >> > > For the flexibility, and all the integrity features of ZFS. The pooled > storage concept is just so much nicer/easier to work with than hardware RAID > arrays, separate LUNs, separate volume managers, separate partitions, etc. > > Need more storage? Just add another raidz vdev to the pool. Instantly have > more storage space, and performance increases as well (the pool stripes > across all the vdevs by default). Don't have any more drive bays? Then > just replace the drives in the raidz vdev with larger ones. All the space > becomes available to the pool. And *all* the filesystems use that pool, so > they all get access to the extra space (no reformatting, no repartitioning, > no offline expansion required). > > Add in the snapshots feature, that actually works without slowing down the > system (UFS) or requiring "wasted"/used space (LVM), and it's hard to use > hardware RAID anymore. :) > > Or course, we do still use hardware RAID controllers, for the disk > management and alerting features, the onboard cache, the fast buses > (PCI-X/PCIe), multi-lane cabling, hot-plug support, etc; we just don't use > the actual RAID features. > > All of our Linux servers still use hardware RAID (5 and 10), with LVM on > top, and XFS on top of that. But it's just not as nice of a storage stack > to work with. :) > > Any special thing to do to make it work with heartbeat? Tonino -- ------------------------------------------------------------ Inter@zioni Interazioni di Antonio Nati http://www.interazioni.it tonix@... ------------------------------------------------------------ _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
|
|
|
Re: just a quickie please
by Shannon Wheeler
::
Rate this Message:
I just tried another users account and it's able to send. In her case, both
her username and e-mail name are the same. I do use aliases and redirection for many accounts... after all, I can't have 20 users named 'sales'. The big difference with this one just seemed to be that FreeBSD complained about an 18 character username. Sendmail doesn't seem to have a problem with it though. >> Is mcaonline.ca in local-host-names? > > yes. And other mcaonline.ca users can send... I better go check whether > any > use webmail... > > Thank you, > Shannon Wheeler > McMurray Computer Experts > 780-790-0728 > > > ----- Original Message ----- > From: "Ekkehard Gehm" <gehm@...> > To: "Shannon Wheeler" <swheeler@...> > Sent: Wednesday, July 08, 2009 12:18 PM > Subject: Re: just a quickie please > > >> Shannon Wheeler schrieb: >>> I'm not currently subscribed to a sendmail list and >>> I'm leaving for vacation tomorrow so >>> of course a problem will show up today. >>> >>> FreeBSD 6ish >>> sendmail 13 >>> SquirrelMail 1.4.13 >>> >>> If I log into squirrelmail, I can send mail no problem. >>> >>> If user is_practitioner logs in, when sending mail she receives: >>> >>> ERROR: >>> Message not sent. Server replied: >>> Requested action not taken: mailbox unavailable >>> 550 5.1.1 <is_practitioner@...>... I don't know who >>> you're looking for. >>> >>> ~~~~~~~~~~~~~~ >>> Note that the user's logon name is: is_practitioner >>> but her e-mail address is: isssp_practitioner@... >>> (FreeBSD limited me to 16 characters for the username) >>> >>> note also that the reject message lists the main domain that the machine >>> belongs to but her e-mail address is a seperate virtual domain. >>> >>> When setting up this user, I set her shell as 'mail-only' even though no >>> such shell exists... most of my other users are set for 'nologin'. >>> In fact, hardly any users use the webmail, especially not for sending. >>> But it should work and does work for other users that I've tried. >>> >>> Anyone know exactly what the problem is? otherwise, just where to >>> look... >>> username vs alias problem? >>> mail-only shell problem? >>> something else? >>> >>> Thank you, >>> Shannon Wheeler >>> 780-790-0728 >>> _______________________________________________ >>> freebsd-isp@... mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-isp >>> To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." >>> >> >> > > _______________________________________________ > freebsd-isp@... mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-isp > To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
|
|
Re: just a quickie please
by Shannon Wheeler
::
Rate this Message:
>>>> If user is_practitioner logs in, when sending mail she receives:
>>>> >>>> ERROR: >>>> Message not sent. Server replied: >>>> Requested action not taken: mailbox unavailable >>>> 550 5.1.1 <is_practitioner@...>... I don't know who >>>> you're looking for. Stupid me. I finally remembered that when you first log into an account on this squirrelmail system you have to go into 'options' and setup your actual e-mail address. thanks folks, Shannon _______________________________________________ freebsd-isp@... mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@..." |
smime.p7s (4K) 
| Free embeddable forum powered by Nabble | Forum Help |
