tag:old.nabble.com,2006:forum-6580 2009-12-24T08:53:29Z Filesystems tag:old.nabble.com,2006:post-26915619 2009-12-24T08:53:29Z 2009-12-24T08:53:29Z Ronald Klop-2 Isn't it write caching? <br>My Solaris machine at work also flushes the data every 30 seconds. <br><br>Ronald. <br><br>On Wed, 23 Dec 2009 21:26:12 +0100, Solon Lutz &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26915619&i=0" target="_top" rel="nofollow">solon@...</a>&gt; wrote: <br><div class='shrinkable-quote'><br>&gt; Hi, <br>&gt; <br>&gt; I opted for two 12-disc raidz2. <br>&gt; Reasons were: Space is more important than performance. <br>&gt; <br>&gt; But performance is very poor - have a look at the iostats - sometimes &nbsp; <br>&gt; nothing really seems <br>&gt; to happen for up to ten seconds, or very little data gets &nbsp;written. &nbsp; <br>&gt; Might this be a problem <br>&gt; of the amd64 system having only 4GB of RAM? Any tuneable sysctls? &nbsp; <br>&gt; Enabling prefetch didn't help...: <br>&gt; <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;capacity &nbsp; &nbsp; operations &nbsp; &nbsp;bandwidth &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <br>&gt; capacity &nbsp; &nbsp; operations &nbsp; &nbsp;bandwidth <br>&gt; pool &nbsp; &nbsp; &nbsp; &nbsp; used &nbsp;avail &nbsp; read &nbsp;write &nbsp; read &nbsp;write &nbsp; pool &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <br>&gt; used &nbsp;avail &nbsp; read &nbsp;write &nbsp; read &nbsp;write <br>&gt; ---------- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp; ---------- &nbsp; <br>&gt; ----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp;1.23K &nbsp; &nbsp; &nbsp;0 &nbsp; 153M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;510 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;510 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;835 &nbsp; &nbsp; &nbsp;0 &nbsp; 104M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;645 &nbsp; &nbsp; &nbsp;0 &nbsp;68.7M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;474 &nbsp; &nbsp; &nbsp;0 &nbsp;59.3M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;5.49K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;494 &nbsp; &nbsp; &nbsp;0 &nbsp;61.8M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 53 &nbsp; &nbsp; &nbsp;0 &nbsp;6.73M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;0 &nbsp;39.4K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 30 &nbsp; &nbsp; &nbsp;0 &nbsp; 140K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 33 &nbsp; &nbsp; &nbsp;0 &nbsp;72.8K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 65 &nbsp; &nbsp; &nbsp;0 &nbsp; 162K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 71 &nbsp; &nbsp; &nbsp;0 &nbsp;8.92M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;636 &nbsp; &nbsp; &nbsp;0 &nbsp;79.4M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;10.5K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;485 &nbsp; &nbsp; &nbsp;0 &nbsp;60.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 85 &nbsp; &nbsp; &nbsp;0 &nbsp;10.7M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 15 &nbsp; &nbsp; &nbsp;0 &nbsp;39.9K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 17 &nbsp; &nbsp; &nbsp;0 &nbsp;40.4K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 31 &nbsp; &nbsp; &nbsp;0 &nbsp;73.3K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 62 &nbsp; &nbsp; &nbsp;0 &nbsp; 156K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 17 &nbsp; &nbsp; &nbsp;0 &nbsp;2.24M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;189 &nbsp; &nbsp; &nbsp;0 &nbsp;23.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;9 &nbsp; &nbsp; &nbsp;0 &nbsp;14.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; 74 &nbsp; 133K &nbsp; 247K store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 12 &nbsp; &nbsp; &nbsp;0 &nbsp;17.5K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; 255K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;100 &nbsp; &nbsp; &nbsp;0 &nbsp;12.6M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; 1012 &nbsp; &nbsp; &nbsp;0 &nbsp; 126M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 33 &nbsp; &nbsp; &nbsp;0 &nbsp; 137K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;0 &nbsp; 293K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;153 &nbsp; &nbsp; &nbsp;0 &nbsp;19.2M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;109 &nbsp; &nbsp; &nbsp;0 &nbsp;13.7M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;15.5K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp;1.54K &nbsp; &nbsp; &nbsp;0 &nbsp; 188M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;0 &nbsp; 888K &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;556 &nbsp; &nbsp; &nbsp;0 &nbsp;69.5M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp;11.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;500 &nbsp; &nbsp; &nbsp;0 &nbsp;62.5M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;0 &nbsp; 766K &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 16 &nbsp; &nbsp; &nbsp;0 &nbsp;30.4K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 29 &nbsp; &nbsp; &nbsp;0 &nbsp; 136K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 38 &nbsp; &nbsp; &nbsp;0 &nbsp;84.7K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 57 &nbsp; &nbsp; &nbsp;0 &nbsp; 145K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 92 &nbsp; &nbsp; &nbsp;0 &nbsp;11.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;497 &nbsp; &nbsp; &nbsp;0 &nbsp;62.1M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp;9.48K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;534 &nbsp; &nbsp; &nbsp;0 &nbsp;66.7M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;153 &nbsp; &nbsp; &nbsp;0 &nbsp;19.2M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 12 &nbsp; &nbsp; &nbsp;0 &nbsp;12.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 26 &nbsp; &nbsp; &nbsp;0 &nbsp; 101K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 34 &nbsp; &nbsp; &nbsp;0 &nbsp;76.3K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 62 &nbsp; &nbsp; &nbsp;0 &nbsp; 155K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 75 &nbsp; &nbsp; &nbsp;0 &nbsp;9.48M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;690 &nbsp; &nbsp; &nbsp;0 &nbsp;86.3M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;0 &nbsp;12.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp;0 &nbsp;14.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; 255K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp; <br>&gt; 15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; <br>&gt; <br>&gt; <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26915619&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26915619&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; </div><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26915619&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26915619&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26914818 2009-12-24T07:17:55Z 2009-12-24T07:17:55Z linimon Old Synopsis: [lor] ufs/unionfs(/ufs) <br>New Synopsis: [unionfs] [lor] ufs/unionfs(/ufs) <br><br>Responsible-Changed-From-To: freebsd-bugs-&gt;freebsd-fs <br>Responsible-Changed-By: linimon <br>Responsible-Changed-When: Thu Dec 24 15:17:40 UTC 2009 <br>Responsible-Changed-Why: <br>Over to maintainer(s). <br><br><a href="http://www.freebsd.org/cgi/query-pr.cgi?pr=141950" target="_top" rel="nofollow">http://www.freebsd.org/cgi/query-pr.cgi?pr=141950</a><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26914818&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26914818&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26913805 2009-12-24T05:17:23Z 2009-12-24T05:17:23Z Wes Morgan-2 On Wed, 23 Dec 2009, Steven Schlansker wrote: <br><div class='shrinkable-quote'><br>&gt; <br>&gt; On Dec 22, 2009, at 5:41 PM, Rich wrote: <br>&gt; <br>&gt;&gt; <a href="http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763" target="_top" rel="nofollow">http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763</a>&nbsp;may be <br>&gt;&gt; useful to you - it's what we did when we got stuck in a resilver loop. <br>&gt;&gt; I recall being in the same state you're in right now at one point, and <br>&gt;&gt; getting out of it from there. <br>&gt;&gt; <br>&gt;&gt; I think if you apply that patch, you'll be able to cancel the <br>&gt;&gt; resilver, and then resilver again with the device you'd like to <br>&gt;&gt; resilver with. <br>&gt;&gt; <br>&gt; <br>&gt; Thanks for the suggestion, but the problem isn't that it's stuck <br>&gt; in a resilver loop (which is what the patch seems to try to avoid) <br>&gt; but that I can't detach a drive. <br>&gt; <br>&gt; Now I got clever and fudged a label onto the new drive (copied the first <br>&gt; 50MB of one of the dying drives), ran a scrub, and have this layout - <br>&gt; <br>&gt; &nbsp;pool: universe <br>&gt; state: DEGRADED <br>&gt; status: One or more devices has experienced an unrecoverable error. &nbsp;An <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp;attempt was made to correct the error. &nbsp;Applications are unaffected. <br>&gt; action: Determine if the device needs to be replaced, and clear the errors <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp;using 'zpool clear' or replace the device with 'zpool replace'. <br>&gt; &nbsp; see: <a href="http://www.sun.com/msg/ZFS-8000-9P" target="_top" rel="nofollow">http://www.sun.com/msg/ZFS-8000-9P</a><br>&gt; scrub: scrub completed after 20h58m with 0 errors on Wed Dec 23 11:36:43 2009 <br>&gt; config: <br>&gt; <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp;NAME &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; STATE &nbsp; &nbsp; READ WRITE CKSUM <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp;universe &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;raidz2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad16 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;replacing &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 40.7M <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad26 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp;506G repaired <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;6170688083648327969 &nbsp;UNAVAIL &nbsp; &nbsp; &nbsp;0 88.7M &nbsp; &nbsp; 0 &nbsp;was /dev/ad12 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;concat/back2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad10 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;concat/ad4ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad24 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;concat/ad6ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp;48 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp;28.5K repaired <br>&gt; <br>&gt; Why has the replacing vdev not gone away? &nbsp;I still can't detach - <br>&gt; [steven@universe:~]% sudo zpool detach universe 6170688083648327969 <br>&gt; cannot detach 6170688083648327969: no valid replicas <br>&gt; even though now there actually is a valid replica (ad26) </div><br>Try detaching ad26. If it lets you do that it will abort the replacement <br>and then you just do another replacement with the real device. If it won't <br>let you do that, you may be stuck having to do some metadata <br>tricks. <br><br><div class='shrinkable-quote'><br>&gt; Additionally, running zpool clear hangs permanently and in fact freezes all IO <br>&gt; to the pool. &nbsp;Since I've mounted /usr from the pool, this is effectively <br>&gt; death to the system. &nbsp;Any other zfs commands seem to work okay <br>&gt; (zpool scrub, zfs mount, etc.). &nbsp;Just clear is insta-death. &nbsp;I can't <br>&gt; help but suspect that this is caused by the now non-sensical vdev configuration <br>&gt; (replacing with one good drive and one nonexistent one)... <br>&gt; <br>&gt; Any further thoughts? &nbsp;Thanks, <br>&gt; Steven <br>&gt; <br>&gt; <br>&gt;&gt; - Rich <br>&gt;&gt; <br>&gt;&gt; On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=0" target="_top" rel="nofollow">000.fbsd@...</a>&gt; wrote: <br>&gt;&gt;&gt; Steven Schlansker wrote: <br>&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt;&gt;&gt; This is because I have drives which are very slightly different in size (&lt; <br>&gt;&gt;&gt;&gt; &nbsp;1MB) <br>&gt;&gt;&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully <br>&gt;&gt;&gt;&gt; (?) <br>&gt;&gt;&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt;&gt;&gt; This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>&gt;&gt;&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a <br>&gt;&gt;&gt;&gt; time <br>&gt;&gt;&gt;&gt; using gconcat and space on another drive! &nbsp;Very frustrating... <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; You can avoid it by partitioning the drives to the well known 'minimal' size <br>&gt;&gt;&gt; (size of smallest disk) and use the partition instead of raw disk. <br>&gt;&gt;&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt;&gt;&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; You can also use labels instead of device name. <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; Miroslav Lachman <br>&gt;&gt;&gt; _______________________________________________ <br>&gt;&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt;&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; -- <br>&gt;&gt; <br>&gt;&gt; If you are over 80 years old and accompanied by your parents, we will <br>&gt;&gt; cash your check. <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; </div>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=5" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26913805&i=6" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26910621 2009-12-23T21:05:46Z 2009-12-23T21:05:46Z Thomas Burgess On Wed, Dec 23, 2009 at 10:20 PM, Rich &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910621&i=0" target="_top" rel="nofollow">rincebrain@...</a>&gt; wrote: <br><br>&gt; FreeBSD 7 or 8? Because 8 purports to need far less tuning... <br>&gt; <br>&gt; - Rich <br>&gt; <br>&gt; That only applies to amd64. &nbsp;i386 still needs tuning. <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910621&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910621&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26910181 2009-12-23T19:20:33Z 2009-12-23T19:20:33Z Rich-105 FreeBSD 7 or 8? Because 8 purports to need far less tuning... <br><br>- Rich <br><br>On Wed, Dec 23, 2009 at 10:18 PM, Matt Simerson &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=0" target="_top" rel="nofollow">matt@...</a>&gt; wrote: <div class='shrinkable-quote'><div class='shrinkable-quote'><br>&gt; <br>&gt; On Dec 23, 2009, at 12:26 PM, Solon Lutz wrote: <br>&gt; <br>&gt;&gt; Hi, <br>&gt;&gt; <br>&gt;&gt; I opted for two 12-disc raidz2. <br>&gt;&gt; Reasons were: Space is more important than performance. <br>&gt;&gt; <br>&gt;&gt; But performance is very poor - have a look at the iostats - sometimes nothing really seems <br>&gt;&gt; to happen for up to ten seconds, or very little data gets  written. Might this be a problem <br>&gt;&gt; of the amd64 system having only 4GB of RAM? Any tuneable sysctls? Enabling prefetch didn't help...: <br>&gt; <br>&gt; When testing on a i386 system with only 4GB of ram, disabling prefetch did help. There are the settings I used, way back when... <br>&gt; <br>&gt; $ cat /boot/loader.conf <br>&gt; #vm.kmem_size=&quot;1024M&quot; <br>&gt; #vm.kmem_size_max=&quot;1024M&quot; <br>&gt; #vfs.zfs.prefetch_disable=1 <br>&gt; #vfs.zfs.arc_min=&quot;16M&quot; <br>&gt; #vfs.zfs.arc_max=&quot;64M&quot; <br>&gt; <br>&gt; I ultimately worked around that problem by no longer using raidz2. <br>&gt; <br>&gt; Matt <br>&gt; <br>&gt; <br>&gt;&gt;               capacity     operations    bandwidth                   capacity     operations    bandwidth <br>&gt;&gt; pool         used  avail   read  write   read  write   pool         used  avail   read  write   read  write <br>&gt;&gt; ----------  -----  -----  -----  -----  -----  -----   ----------  -----  -----  -----  -----  -----  ----- <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0  1.23K      0   153M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0    510 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0    510 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0    835      0   104M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0    645      0  68.7M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      616G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    474      0  59.3M      0  store-1      616G  15.6T      0      4      0  5.49K <br>&gt;&gt; backup-7    1.23T   130G    494      0  61.8M      0  store-1      616G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G     53      0  6.73M      0  store-1      617G  15.6T      0     19      0  39.4K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     30      0   140K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     33      0  72.8K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     65      0   162K <br>&gt;&gt; backup-7    1.23T   130G     71      0  8.92M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    636      0  79.4M      0  store-1      617G  15.6T      0      4      0  10.5K <br>&gt;&gt; backup-7    1.23T   130G    485      0  60.6M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G     85      0  10.7M      0  store-1      617G  15.6T      0     15      0  39.9K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     17      0  40.4K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     31      0  73.3K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     62      0   156K <br>&gt;&gt; backup-7    1.23T   130G     17      0  2.24M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    189      0  23.6M      0  store-1      617G  15.6T      0      9      0  14.0K <br>&gt;&gt; backup-7    1.23T   130G      2     74   133K   247K  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     12      0  17.5K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      1      0   255K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0    100      0  12.6M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0   1012      0   126M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     33      0   137K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     19      0   293K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0    153      0  19.2M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0    109      0  13.7M <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      4      0  15.5K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0  1.54K      0   188M <br>&gt;&gt; backup-7    1.23T   130G      6      0   888K      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    556      0  69.5M      0  store-1      617G  15.6T      0      8      0  11.0K <br>&gt;&gt; backup-7    1.23T   130G    500      0  62.5M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      5      0   766K      0  store-1      617G  15.6T      0     16      0  30.4K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     29      0   136K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     38      0  84.7K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     57      0   145K <br>&gt;&gt; backup-7    1.23T   130G     92      0  11.6M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    497      0  62.1M      0  store-1      617G  15.6T      0      8      0  9.48K <br>&gt;&gt; backup-7    1.23T   130G    534      0  66.7M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    153      0  19.2M      0  store-1      617G  15.6T      0     12      0  12.0K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     26      0   101K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     34      0  76.3K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     62      0   155K <br>&gt;&gt; backup-7    1.23T   130G     75      0  9.48M      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G    690      0  86.3M      0  store-1      617G  15.6T      0      6      0  12.0K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0     10      0  14.0K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      1      0   255K <br>&gt;&gt; backup-7    1.23T   130G      0      0      0      0  store-1      617G  15.6T      0      0      0      0 <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; _______________________________________________ <br>&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; </div></div><br><br>-- <br><br>Prefiro perder a guerra e ganhar a paz. -- Bob Marley <br><br />_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=5" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910181&i=6" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26910160 2009-12-23T19:16:16Z 2009-12-23T19:16:16Z Matt Simerson-3 <br>On Dec 23, 2009, at 12:26 PM, Solon Lutz wrote: <br><br>&gt; Hi, <br>&gt; <br>&gt; I opted for two 12-disc raidz2. <br>&gt; Reasons were: Space is more important than performance. <br>&gt; <br>&gt; But performance is very poor - have a look at the iostats - sometimes nothing really seems <br>&gt; to happen for up to ten seconds, or very little data gets &nbsp;written. Might this be a problem <br>&gt; of the amd64 system having only 4GB of RAM? Any tuneable sysctls? Enabling prefetch didn't help...: <br><br>When testing on a i386 system with only 4GB of ram, disabling prefetch did help. There are the settings I used, way back when... <br><br>$ cat /boot/loader.conf <br>#vm.kmem_size=&quot;1024M&quot; <br>#vm.kmem_size_max=&quot;1024M&quot; <br>#vfs.zfs.prefetch_disable=1 <br>#vfs.zfs.arc_min=&quot;16M&quot; <br>#vfs.zfs.arc_max=&quot;64M&quot; <br><br>I ultimately worked around that problem by no longer using raidz2. <br><br>Matt <br><br><div class='shrinkable-quote'><br>&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; capacity &nbsp; &nbsp; operations &nbsp; &nbsp;bandwidth &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; capacity &nbsp; &nbsp; operations &nbsp; &nbsp;bandwidth <br>&gt; pool &nbsp; &nbsp; &nbsp; &nbsp; used &nbsp;avail &nbsp; read &nbsp;write &nbsp; read &nbsp;write &nbsp; pool &nbsp; &nbsp; &nbsp; &nbsp; used &nbsp;avail &nbsp; read &nbsp;write &nbsp; read &nbsp;write <br>&gt; ---------- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp; ---------- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp;1.23K &nbsp; &nbsp; &nbsp;0 &nbsp; 153M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;510 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;510 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;835 &nbsp; &nbsp; &nbsp;0 &nbsp; 104M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;645 &nbsp; &nbsp; &nbsp;0 &nbsp;68.7M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;474 &nbsp; &nbsp; &nbsp;0 &nbsp;59.3M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;5.49K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;494 &nbsp; &nbsp; &nbsp;0 &nbsp;61.8M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 53 &nbsp; &nbsp; &nbsp;0 &nbsp;6.73M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;0 &nbsp;39.4K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 30 &nbsp; &nbsp; &nbsp;0 &nbsp; 140K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 33 &nbsp; &nbsp; &nbsp;0 &nbsp;72.8K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 65 &nbsp; &nbsp; &nbsp;0 &nbsp; 162K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 71 &nbsp; &nbsp; &nbsp;0 &nbsp;8.92M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;636 &nbsp; &nbsp; &nbsp;0 &nbsp;79.4M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;10.5K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;485 &nbsp; &nbsp; &nbsp;0 &nbsp;60.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 85 &nbsp; &nbsp; &nbsp;0 &nbsp;10.7M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 15 &nbsp; &nbsp; &nbsp;0 &nbsp;39.9K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 17 &nbsp; &nbsp; &nbsp;0 &nbsp;40.4K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 31 &nbsp; &nbsp; &nbsp;0 &nbsp;73.3K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 62 &nbsp; &nbsp; &nbsp;0 &nbsp; 156K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 17 &nbsp; &nbsp; &nbsp;0 &nbsp;2.24M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;189 &nbsp; &nbsp; &nbsp;0 &nbsp;23.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;9 &nbsp; &nbsp; &nbsp;0 &nbsp;14.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; 74 &nbsp; 133K &nbsp; 247K store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 12 &nbsp; &nbsp; &nbsp;0 &nbsp;17.5K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; 255K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;100 &nbsp; &nbsp; &nbsp;0 &nbsp;12.6M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; 1012 &nbsp; &nbsp; &nbsp;0 &nbsp; 126M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 33 &nbsp; &nbsp; &nbsp;0 &nbsp; 137K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;0 &nbsp; 293K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;153 &nbsp; &nbsp; &nbsp;0 &nbsp;19.2M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;109 &nbsp; &nbsp; &nbsp;0 &nbsp;13.7M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;15.5K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp;1.54K &nbsp; &nbsp; &nbsp;0 &nbsp; 188M <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;0 &nbsp; 888K &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;556 &nbsp; &nbsp; &nbsp;0 &nbsp;69.5M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp;11.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;500 &nbsp; &nbsp; &nbsp;0 &nbsp;62.5M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;0 &nbsp; 766K &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 16 &nbsp; &nbsp; &nbsp;0 &nbsp;30.4K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 29 &nbsp; &nbsp; &nbsp;0 &nbsp; 136K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 38 &nbsp; &nbsp; &nbsp;0 &nbsp;84.7K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 57 &nbsp; &nbsp; &nbsp;0 &nbsp; 145K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 92 &nbsp; &nbsp; &nbsp;0 &nbsp;11.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;497 &nbsp; &nbsp; &nbsp;0 &nbsp;62.1M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp;9.48K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;534 &nbsp; &nbsp; &nbsp;0 &nbsp;66.7M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;153 &nbsp; &nbsp; &nbsp;0 &nbsp;19.2M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 12 &nbsp; &nbsp; &nbsp;0 &nbsp;12.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 26 &nbsp; &nbsp; &nbsp;0 &nbsp; 101K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 34 &nbsp; &nbsp; &nbsp;0 &nbsp;76.3K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 62 &nbsp; &nbsp; &nbsp;0 &nbsp; 155K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 75 &nbsp; &nbsp; &nbsp;0 &nbsp;9.48M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;690 &nbsp; &nbsp; &nbsp;0 &nbsp;86.3M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;0 &nbsp;12.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp;0 &nbsp;14.0K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; 255K <br>&gt; backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>&gt; <br>&gt; <br>&gt; <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910160&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910160&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; </div><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910160&i=2" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26910160&i=3" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909579 2009-12-23T17:30:13Z 2009-12-23T17:30:13Z Rich-105 IPMI cards require a slot to go in, but I believe you can buy PCI <br>cards that are just an IPMI slot. <br><br>It can't be &quot;that hard&quot;...you just flush disk state and then triple-fault... <br><br>- Rich <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909579&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909579&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909556 2009-12-23T17:25:06Z 2009-12-23T17:25:06Z Steven Schlansker-4 <br>On Dec 23, 2009, at 5:06 PM, Rich wrote: <br><br>&gt; IPMI is a great thing, let me tell you. :) <br><br>Can one just buy a random card and plug it in to a generic <br>motherboard and have it work? &nbsp;I was under the impression <br>that this was generally only available on server-class <br>hardware... <br><br>&gt; <br>&gt; Also, Linux's magic sysrq has saved me several times. <br><br><br>Me too! &nbsp;I wish FreeBSD had something like this... maybe I'll <br>write a kernel patch to support just the hard-reset <br>bit of it... ;-) <br><br><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909556&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909556&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909453 2009-12-23T17:06:13Z 2009-12-23T17:06:13Z Rich-105 IPMI is a great thing, let me tell you. :) <br><br>Also, Linux's magic sysrq has saved me several times. <br><br>- Rich <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909453&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909453&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909439 2009-12-23T17:04:30Z 2009-12-23T17:04:30Z Steven Schlansker-4 <br>On Dec 23, 2009, at 5:02 PM, Rich wrote: <br><br>&gt; I mean, the report in that link I sent you was A) mine and B) on 13TB <br>&gt; of data I couldn't lose, so I can sympathize with the problem... <br>&gt; <br>&gt; A dirty solution might be to see if Solaris does the right thing with <br>&gt; ejecting the bad drive. <br><br>That's not a bad plan. &nbsp;I've got a Solaris USB stick to boot off of, <br>but it'll be a few days at least before I am within USB range of <br>the server... :-p <br><br>I do wish commodity hardware had remote reset functionality! &nbsp;Having <br>a server crash remotely is so frustrating... <br><br><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909439&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909439&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909422 2009-12-23T17:02:31Z 2009-12-23T17:02:31Z Steven Schlansker-4 <br><br>On Dec 23, 2009, at 4:32 PM, Rich wrote: <br><br>&gt; That's fascinating - I'd swear it used to be the case (in <br>&gt; Solaris-land, at least) that resilvering with a smaller vdev resulted <br>&gt; in it shrinking the available space on other vdevs as though they were <br>&gt; all as large as the smallest vdev available. <br><br>Pretty sure that this doesn't exist for raidz. &nbsp;I haven't tried, though, <br>and Sun's bug database's search blows chunks. &nbsp;I remember seeing <br>a bug filed on it before, but I can't for the life of me find it. <br><br>&gt; <br>&gt; In particular, I'd swear I've done this with some disk arrays I have <br>&gt; laying around with 7x removable SCA drives, which I have in 2, 4.5, 9, <br>&gt; and 18 GB varieties... <br>&gt; <br>&gt; But maybe I'm just hallucinating, or this went away a long time ago. <br>&gt; (This was circa b70 in Solaris.) <br><br>Shrinking of mirrored drives seems like it might be working. <br>Again Sun's bug database isn't clear at all about what can / <br>can't be shrunk - maybe I should get a Solaris bootdisk and see <br>if I can shrink it from there... <br><div class='shrinkable-quote'><br>&gt; <br>&gt; I know you can't do this in FreeBSD; I've also run into the <br>&gt; &quot;insufficient space&quot; problem when trying to replace with a smaller <br>&gt; vdev. <br>&gt; <br>&gt; - Rich <br>&gt; <br>&gt; On Wed, Dec 23, 2009 at 7:29 PM, Steven Schlansker <br>&gt; &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909422&i=0" target="_top" rel="nofollow">stevenschlansker@...</a>&gt; wrote: <br>&gt;&gt; <br>&gt;&gt; On Dec 22, 2009, at 3:15 PM, Miroslav Lachman wrote: <br>&gt;&gt; <br>&gt;&gt;&gt; Steven Schlansker wrote: <br>&gt;&gt;&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt;&gt;&gt; This is because I have drives which are very slightly different in size (&lt; &nbsp;1MB) <br>&gt;&gt;&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully (?) <br>&gt;&gt;&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt;&gt;&gt; This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>&gt;&gt;&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a time <br>&gt;&gt;&gt;&gt; using gconcat and space on another drive! &nbsp;Very frustrating... <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; You can avoid it by partitioning the drives to the well known 'minimal' size (size of smallest disk) and use the partition instead of raw disk. <br>&gt;&gt;&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt;&gt;&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; Yes, this makes sense. &nbsp;Unfortunately, I didn't do this when I first made the array <br>&gt;&gt; as the documentation says you should use whole disks so that it can enable the write <br>&gt;&gt; cache, which I took to mean you shouldn't use a partition table. &nbsp;And now there's no <br>&gt;&gt; way to fix it after the fact, as you can't shrink a zpool even by a single <br>&gt;&gt; MB :( <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; _______________________________________________ <br>&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909422&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909422&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt;&gt; <br>&gt; <br>&gt; <br>&gt; <br>&gt; -- <br>&gt; <br>&gt; [We] use bad software and bad machines for the wrong things. -- R. W. Hamming </div><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909422&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909422&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909421 2009-12-23T17:02:31Z 2009-12-23T17:02:31Z Rich-105 I mean, the report in that link I sent you was A) mine and B) on 13TB <br>of data I couldn't lose, so I can sympathize with the problem... <br><br>A dirty solution might be to see if Solaris does the right thing with <br>ejecting the bad drive. <br><br>- Rich <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909421&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909421&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909390 2009-12-23T16:57:12Z 2009-12-23T16:57:12Z Steven Schlansker-4 On Dec 23, 2009, at 4:44 PM, Rich wrote: <br><br>&gt; Export then import, perhaps? <br>&gt; <br><br>Sadly exporting seems to write the state of this <br>funky &quot;replacing&quot; device out along with everything <br>else, so importing just brings it back with. <br>Granted I haven't tried this in the new state with the <br>newly-resilvered drive, but I don't hold much hope. <br><br>I'll try it next time I'm physically with my server, as <br>I now have to reboot it anyway (sadness!) <br><br><br>&gt; I don't honestly know what to suggest - there are horrid workarounds <br>&gt; you could do involving manually diddling the metadata state, but I <br>&gt; feel like the correct solution is to open up a bug report and get a <br>&gt; fix put in. <br>&gt; <br><br>I'd be more willing to try hackish workarounds if I <br>either didn't care about the data or had proper backups... <br>but I don't so I'm rather worried about trashing the <br>pool. &nbsp;It's so hard to back up 6TB of data on a college <br>kid's budget! &nbsp;:( <br><br>I'd file a PR but the last three times I've filed things on the <br>FreeBSD bug tracker, they've gone largely ignored. &nbsp;One from two <br>years ago still is open, and one from last summer hasn't even <br>been replied to... so I've rather given up on it. <br><div class='shrinkable-quote'><br>&gt; - Rich <br>&gt; <br>&gt; On Wed, Dec 23, 2009 at 7:36 PM, Steven Schlansker <br>&gt; &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=0" target="_top" rel="nofollow">stevenschlansker@...</a>&gt; wrote: <br>&gt;&gt; <br>&gt;&gt; On Dec 22, 2009, at 5:41 PM, Rich wrote: <br>&gt;&gt; <br>&gt;&gt;&gt; <a href="http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763" target="_top" rel="nofollow">http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763</a>&nbsp;may be <br>&gt;&gt;&gt; useful to you - it's what we did when we got stuck in a resilver loop. <br>&gt;&gt;&gt; I recall being in the same state you're in right now at one point, and <br>&gt;&gt;&gt; getting out of it from there. <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; I think if you apply that patch, you'll be able to cancel the <br>&gt;&gt;&gt; resilver, and then resilver again with the device you'd like to <br>&gt;&gt;&gt; resilver with. <br>&gt;&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; Thanks for the suggestion, but the problem isn't that it's stuck <br>&gt;&gt; in a resilver loop (which is what the patch seems to try to avoid) <br>&gt;&gt; but that I can't detach a drive. <br>&gt;&gt; <br>&gt;&gt; Now I got clever and fudged a label onto the new drive (copied the first <br>&gt;&gt; 50MB of one of the dying drives), ran a scrub, and have this layout - <br>&gt;&gt; <br>&gt;&gt; &nbsp;pool: universe <br>&gt;&gt; &nbsp;state: DEGRADED <br>&gt;&gt; status: One or more devices has experienced an unrecoverable error. &nbsp;An <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp;attempt was made to correct the error. &nbsp;Applications are unaffected. <br>&gt;&gt; action: Determine if the device needs to be replaced, and clear the errors <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp;using 'zpool clear' or replace the device with 'zpool replace'. <br>&gt;&gt; &nbsp; see: <a href="http://www.sun.com/msg/ZFS-8000-9P" target="_top" rel="nofollow">http://www.sun.com/msg/ZFS-8000-9P</a><br>&gt;&gt; &nbsp;scrub: scrub completed after 20h58m with 0 errors on Wed Dec 23 11:36:43 2009 <br>&gt;&gt; config: <br>&gt;&gt; <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp;NAME &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; STATE &nbsp; &nbsp; READ WRITE CKSUM <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp;universe &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;raidz2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad16 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;replacing &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 40.7M <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad26 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp;506G repaired <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;6170688083648327969 &nbsp;UNAVAIL &nbsp; &nbsp; &nbsp;0 88.7M &nbsp; &nbsp; 0 &nbsp;was /dev/ad12 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;concat/back2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad10 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;concat/ad4ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ad24 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&gt;&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;concat/ad6ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp;48 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp;28.5K repaired <br>&gt;&gt; <br>&gt;&gt; Why has the replacing vdev not gone away? &nbsp;I still can't detach - <br>&gt;&gt; [steven@universe:~]% sudo zpool detach universe 6170688083648327969 <br>&gt;&gt; cannot detach 6170688083648327969: no valid replicas <br>&gt;&gt; even though now there actually is a valid replica (ad26) <br>&gt;&gt; <br>&gt;&gt; Additionally, running zpool clear hangs permanently and in fact freezes all IO <br>&gt;&gt; to the pool. &nbsp;Since I've mounted /usr from the pool, this is effectively <br>&gt;&gt; death to the system. &nbsp;Any other zfs commands seem to work okay <br>&gt;&gt; (zpool scrub, zfs mount, etc.). &nbsp;Just clear is insta-death. &nbsp;I can't <br>&gt;&gt; help but suspect that this is caused by the now non-sensical vdev configuration <br>&gt;&gt; (replacing with one good drive and one nonexistent one)... <br>&gt;&gt; <br>&gt;&gt; Any further thoughts? &nbsp;Thanks, <br>&gt;&gt; Steven <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt;&gt; - Rich <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=1" target="_top" rel="nofollow">000.fbsd@...</a>&gt; wrote: <br>&gt;&gt;&gt;&gt; Steven Schlansker wrote: <br>&gt;&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt;&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt;&gt;&gt;&gt; This is because I have drives which are very slightly different in size (&lt; <br>&gt;&gt;&gt;&gt;&gt; &nbsp;1MB) <br>&gt;&gt;&gt;&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully <br>&gt;&gt;&gt;&gt;&gt; (?) <br>&gt;&gt;&gt;&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt;&gt;&gt;&gt; This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>&gt;&gt;&gt;&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a <br>&gt;&gt;&gt;&gt;&gt; time <br>&gt;&gt;&gt;&gt;&gt; using gconcat and space on another drive! &nbsp;Very frustrating... <br>&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt;&gt; You can avoid it by partitioning the drives to the well known 'minimal' size <br>&gt;&gt;&gt;&gt; (size of smallest disk) and use the partition instead of raw disk. <br>&gt;&gt;&gt;&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt;&gt;&gt;&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt;&gt; You can also use labels instead of device name. <br>&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt;&gt; Miroslav Lachman <br>&gt;&gt;&gt;&gt; _______________________________________________ <br>&gt;&gt;&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=2" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt;&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt;&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=3" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; -- <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; If you are over 80 years old and accompanied by your parents, we will <br>&gt;&gt;&gt; cash your check. <br>&gt;&gt; <br>&gt;&gt; _______________________________________________ <br>&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=4" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=5" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt;&gt; <br>&gt; <br>&gt; <br>&gt; <br>&gt; -- <br>&gt; <br>&gt; Forest fires cause Smokey Bears. </div><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=6" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909390&i=7" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909298 2009-12-23T16:44:07Z 2009-12-23T16:44:07Z Rich-105 Export then import, perhaps? <br><br>I don't honestly know what to suggest - there are horrid workarounds <br>you could do involving manually diddling the metadata state, but I <br>feel like the correct solution is to open up a bug report and get a <br>fix put in. <br><br>- Rich <br><br>On Wed, Dec 23, 2009 at 7:36 PM, Steven Schlansker <br>&lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=0" target="_top" rel="nofollow">stevenschlansker@...</a>&gt; wrote: <div class='shrinkable-quote'><br>&gt; <br>&gt; On Dec 22, 2009, at 5:41 PM, Rich wrote: <br>&gt; <br>&gt;&gt; <a href="http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763" target="_top" rel="nofollow">http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763</a>&nbsp;may be <br>&gt;&gt; useful to you - it's what we did when we got stuck in a resilver loop. <br>&gt;&gt; I recall being in the same state you're in right now at one point, and <br>&gt;&gt; getting out of it from there. <br>&gt;&gt; <br>&gt;&gt; I think if you apply that patch, you'll be able to cancel the <br>&gt;&gt; resilver, and then resilver again with the device you'd like to <br>&gt;&gt; resilver with. <br>&gt;&gt; <br>&gt; <br>&gt; Thanks for the suggestion, but the problem isn't that it's stuck <br>&gt; in a resilver loop (which is what the patch seems to try to avoid) <br>&gt; but that I can't detach a drive. <br>&gt; <br>&gt; Now I got clever and fudged a label onto the new drive (copied the first <br>&gt; 50MB of one of the dying drives), ran a scrub, and have this layout - <br>&gt; <br>&gt;  pool: universe <br>&gt;  state: DEGRADED <br>&gt; status: One or more devices has experienced an unrecoverable error.  An <br>&gt;        attempt was made to correct the error.  Applications are unaffected. <br>&gt; action: Determine if the device needs to be replaced, and clear the errors <br>&gt;        using 'zpool clear' or replace the device with 'zpool replace'. <br>&gt;   see: <a href="http://www.sun.com/msg/ZFS-8000-9P" target="_top" rel="nofollow">http://www.sun.com/msg/ZFS-8000-9P</a><br>&gt;  scrub: scrub completed after 20h58m with 0 errors on Wed Dec 23 11:36:43 2009 <br>&gt; config: <br>&gt; <br>&gt;        NAME                       STATE     READ WRITE CKSUM <br>&gt;        universe                   DEGRADED     0     0     0 <br>&gt;          raidz2                   DEGRADED     0     0     0 <br>&gt;            ad16                   ONLINE       0     0     0 <br>&gt;            replacing              DEGRADED     0     0 40.7M <br>&gt;              ad26                 ONLINE       0     0     0  506G repaired <br>&gt;              6170688083648327969  UNAVAIL      0 88.7M     0  was /dev/ad12 <br>&gt;            ad8                    ONLINE       0     0     0 <br>&gt;            concat/back2           ONLINE       0     0     0 <br>&gt;            ad10                   ONLINE       0     0     0 <br>&gt;            concat/ad4ex           ONLINE       0     0     0 <br>&gt;            ad24                   ONLINE       0     0     0 <br>&gt;            concat/ad6ex           ONLINE      48     0     0  28.5K repaired <br>&gt; <br>&gt; Why has the replacing vdev not gone away?  I still can't detach - <br>&gt; [steven@universe:~]% sudo zpool detach universe 6170688083648327969 <br>&gt; cannot detach 6170688083648327969: no valid replicas <br>&gt; even though now there actually is a valid replica (ad26) <br>&gt; <br>&gt; Additionally, running zpool clear hangs permanently and in fact freezes all IO <br>&gt; to the pool.  Since I've mounted /usr from the pool, this is effectively <br>&gt; death to the system.  Any other zfs commands seem to work okay <br>&gt; (zpool scrub, zfs mount, etc.).  Just clear is insta-death.  I can't <br>&gt; help but suspect that this is caused by the now non-sensical vdev configuration <br>&gt; (replacing with one good drive and one nonexistent one)... <br>&gt; <br>&gt; Any further thoughts?  Thanks, <br>&gt; Steven <br>&gt; <br>&gt; <br>&gt;&gt; - Rich <br>&gt;&gt; <br>&gt;&gt; On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=1" target="_top" rel="nofollow">000.fbsd@...</a>&gt; wrote: <br>&gt;&gt;&gt; Steven Schlansker wrote: <br>&gt;&gt;&gt;&gt; <br>&gt;&gt;&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt;&gt;&gt; This is because I have drives which are very slightly different in size (&lt; <br>&gt;&gt;&gt;&gt;  1MB) <br>&gt;&gt;&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully <br>&gt;&gt;&gt;&gt; (?) <br>&gt;&gt;&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt;&gt;&gt; This is extremely frustrating... is there any way to fix that?  I'm <br>&gt;&gt;&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a <br>&gt;&gt;&gt;&gt; time <br>&gt;&gt;&gt;&gt; using gconcat and space on another drive!  Very frustrating... <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; You can avoid it by partitioning the drives to the well known 'minimal' size <br>&gt;&gt;&gt; (size of smallest disk) and use the partition instead of raw disk. <br>&gt;&gt;&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt;&gt;&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; You can also use labels instead of device name. <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; Miroslav Lachman <br>&gt;&gt;&gt; _______________________________________________ <br>&gt;&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=2" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=3" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt;&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; <br>&gt;&gt; -- <br>&gt;&gt; <br>&gt;&gt; If you are over 80 years old and accompanied by your parents, we will <br>&gt;&gt; cash your check. <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=4" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=5" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; </div><br><br><br>-- <br><br>Forest fires cause Smokey Bears. <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=6" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909298&i=7" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909244 2009-12-23T16:36:00Z 2009-12-23T16:36:00Z Steven Schlansker-4 <br>On Dec 22, 2009, at 5:41 PM, Rich wrote: <br><br>&gt; <a href="http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763" target="_top" rel="nofollow">http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763</a>&nbsp;may be <br>&gt; useful to you - it's what we did when we got stuck in a resilver loop. <br>&gt; I recall being in the same state you're in right now at one point, and <br>&gt; getting out of it from there. <br>&gt; <br>&gt; I think if you apply that patch, you'll be able to cancel the <br>&gt; resilver, and then resilver again with the device you'd like to <br>&gt; resilver with. <br>&gt; <br><br>Thanks for the suggestion, but the problem isn't that it's stuck <br>in a resilver loop (which is what the patch seems to try to avoid) <br>but that I can't detach a drive. <br><br>Now I got clever and fudged a label onto the new drive (copied the first <br>50MB of one of the dying drives), ran a scrub, and have this layout - <br><br>&nbsp; pool: universe <br>&nbsp;state: DEGRADED <br>status: One or more devices has experienced an unrecoverable error. &nbsp;An <br>&nbsp; &nbsp; &nbsp; &nbsp; attempt was made to correct the error. &nbsp;Applications are unaffected. <br>action: Determine if the device needs to be replaced, and clear the errors <br>&nbsp; &nbsp; &nbsp; &nbsp; using 'zpool clear' or replace the device with 'zpool replace'. <br>&nbsp; &nbsp;see: <a href="http://www.sun.com/msg/ZFS-8000-9P" target="_top" rel="nofollow">http://www.sun.com/msg/ZFS-8000-9P</a><br>&nbsp;scrub: scrub completed after 20h58m with 0 errors on Wed Dec 23 11:36:43 2009 <br>config: <br><br>&nbsp; &nbsp; &nbsp; &nbsp; NAME &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; STATE &nbsp; &nbsp; READ WRITE CKSUM <br>&nbsp; &nbsp; &nbsp; &nbsp; universe &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; raidz2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad16 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; replacing &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 40.7M <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad26 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp;506G repaired <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 6170688083648327969 &nbsp;UNAVAIL &nbsp; &nbsp; &nbsp;0 88.7M &nbsp; &nbsp; 0 &nbsp;was /dev/ad12 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; concat/back2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad10 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; concat/ad4ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad24 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; concat/ad6ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp;48 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp;28.5K repaired <br><br>Why has the replacing vdev not gone away? &nbsp;I still can't detach - <br>[steven@universe:~]% sudo zpool detach universe 6170688083648327969 <br>cannot detach 6170688083648327969: no valid replicas <br>even though now there actually is a valid replica (ad26) <br><br>Additionally, running zpool clear hangs permanently and in fact freezes all IO <br>to the pool. &nbsp;Since I've mounted /usr from the pool, this is effectively <br>death to the system. &nbsp;Any other zfs commands seem to work okay <br>(zpool scrub, zfs mount, etc.). &nbsp;Just clear is insta-death. &nbsp;I can't <br>help but suspect that this is caused by the now non-sensical vdev configuration <br>(replacing with one good drive and one nonexistent one)... <br><br>Any further thoughts? &nbsp;Thanks, <br>Steven <br><br><div class='shrinkable-quote'><br>&gt; - Rich <br>&gt; <br>&gt; On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909244&i=0" target="_top" rel="nofollow">000.fbsd@...</a>&gt; wrote: <br>&gt;&gt; Steven Schlansker wrote: <br>&gt;&gt;&gt; <br>&gt;&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt;&gt; This is because I have drives which are very slightly different in size (&lt; <br>&gt;&gt;&gt; &nbsp;1MB) <br>&gt;&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully <br>&gt;&gt;&gt; (?) <br>&gt;&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt;&gt; This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>&gt;&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a <br>&gt;&gt;&gt; time <br>&gt;&gt;&gt; using gconcat and space on another drive! &nbsp;Very frustrating... <br>&gt;&gt; <br>&gt;&gt; You can avoid it by partitioning the drives to the well known 'minimal' size <br>&gt;&gt; (size of smallest disk) and use the partition instead of raw disk. <br>&gt;&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt;&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt;&gt; <br>&gt;&gt; You can also use labels instead of device name. <br>&gt;&gt; <br>&gt;&gt; Miroslav Lachman <br>&gt;&gt; _______________________________________________ <br>&gt;&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909244&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt;&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt;&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909244&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt;&gt; <br>&gt; <br>&gt; <br>&gt; <br>&gt; -- <br>&gt; <br>&gt; If you are over 80 years old and accompanied by your parents, we will <br>&gt; cash your check. </div><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909244&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909244&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909216 2009-12-23T16:32:53Z 2009-12-23T16:32:53Z Rich-105 That's fascinating - I'd swear it used to be the case (in <br>Solaris-land, at least) that resilvering with a smaller vdev resulted <br>in it shrinking the available space on other vdevs as though they were <br>all as large as the smallest vdev available. <br><br>In particular, I'd swear I've done this with some disk arrays I have <br>laying around with 7x removable SCA drives, which I have in 2, 4.5, 9, <br>and 18 GB varieties... <br><br>But maybe I'm just hallucinating, or this went away a long time ago. <br>(This was circa b70 in Solaris.) <br><br>I know you can't do this in FreeBSD; I've also run into the <br>&quot;insufficient space&quot; problem when trying to replace with a smaller <br>vdev. <br><br>- Rich <br><br>On Wed, Dec 23, 2009 at 7:29 PM, Steven Schlansker <br>&lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909216&i=0" target="_top" rel="nofollow">stevenschlansker@...</a>&gt; wrote: <div class='shrinkable-quote'><br>&gt; <br>&gt; On Dec 22, 2009, at 3:15 PM, Miroslav Lachman wrote: <br>&gt; <br>&gt;&gt; Steven Schlansker wrote: <br>&gt;&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt;&gt; This is because I have drives which are very slightly different in size (&lt;  1MB) <br>&gt;&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully (?) <br>&gt;&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt;&gt; This is extremely frustrating... is there any way to fix that?  I'm <br>&gt;&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a time <br>&gt;&gt;&gt; using gconcat and space on another drive!  Very frustrating... <br>&gt;&gt; <br>&gt;&gt; You can avoid it by partitioning the drives to the well known 'minimal' size (size of smallest disk) and use the partition instead of raw disk. <br>&gt;&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt;&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt; <br>&gt; <br>&gt; Yes, this makes sense.  Unfortunately, I didn't do this when I first made the array <br>&gt; as the documentation says you should use whole disks so that it can enable the write <br>&gt; cache, which I took to mean you shouldn't use a partition table.  And now there's no <br>&gt; way to fix it after the fact, as you can't shrink a zpool even by a single <br>&gt; MB :( <br>&gt; <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909216&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909216&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; </div><br><br><br>-- <br><br>[We] use bad software and bad machines for the wrong things. -- R. W. Hamming <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909216&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909216&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26909185 2009-12-23T16:29:02Z 2009-12-23T16:29:02Z Steven Schlansker-4 <br>On Dec 22, 2009, at 3:15 PM, Miroslav Lachman wrote: <br><div class='shrinkable-quote'><br>&gt; Steven Schlansker wrote: <br>&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt; This is because I have drives which are very slightly different in size (&lt; &nbsp;1MB) <br>&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully (?) <br>&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt; This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a time <br>&gt;&gt; using gconcat and space on another drive! &nbsp;Very frustrating... <br>&gt; <br>&gt; You can avoid it by partitioning the drives to the well known 'minimal' size (size of smallest disk) and use the partition instead of raw disk. <br>&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt; of ad12p1 (if you creat partitions by gpart) </div><br><br>Yes, this makes sense. &nbsp;Unfortunately, I didn't do this when I first made the array <br>as the documentation says you should use whole disks so that it can enable the write <br>cache, which I took to mean you shouldn't use a partition table. &nbsp;And now there's no <br>way to fix it after the fact, as you can't shrink a zpool even by a single <br>MB :( <br><br><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909185&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26909185&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26907668 2009-12-23T13:30:03Z 2009-12-23T13:30:03Z Rusty Nejdl The following reply was made to PR kern/132960; it has been noted by GNATS. <br><br>From: Rusty Nejdl &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907668&i=0" target="_top" rel="nofollow">rnejdl@...</a>&gt; <br>To: <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907668&i=1" target="_top" rel="nofollow">bug-followup@...</a>, <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907668&i=2" target="_top" rel="nofollow">kevinxlinuz@...</a> <br>Cc: &nbsp; <br>Subject: Re: kern/132960: [ufs] [panic] panic:ffs_blkfree: freeing free frag <br>Date: Wed, 23 Dec 2009 14:59:11 -0600 <br><br>&nbsp;--00163649a335302443047b6b989d <br>&nbsp;Content-Type: text/plain; charset=ISO-8859-1 <br>&nbsp; <br>&nbsp;I just saw this on AMD64 on FreeBSD 8.0. &nbsp;I'm kind of surprised to see this <br>&nbsp;still lingering around. &nbsp;Should I continue this problem report or submit a <br>&nbsp;new one? <br>&nbsp; <br>&nbsp;Rusty Nejdl <br>&nbsp; <br>&nbsp;--00163649a335302443047b6b989d <br>&nbsp;Content-Type: text/html; charset=ISO-8859-1 <br>&nbsp;Content-Transfer-Encoding: quoted-printable <br>&nbsp; <br>&nbsp;I just saw this on AMD64 on FreeBSD 8.0.=A0 I&#39;m kind of surprised to se= <br>&nbsp;e this still lingering around.=A0 Should I continue this problem report or = <br>&nbsp;submit a new one?&lt;br&gt;&lt;br&gt;Rusty Nejdl&lt;br&gt; <br>&nbsp; <br>&nbsp;--00163649a335302443047b6b989d-- <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907668&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907668&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26907390 2009-12-23T13:03:59Z 2009-12-23T13:03:59Z Bob Friesenhahn On Wed, 23 Dec 2009, Solon Lutz wrote: <br><br>&gt; I opted for two 12-disc raidz2. <br>&gt; Reasons were: Space is more important than performance. <br>&gt; <br>&gt; But performance is very poor - have a look at the iostats - sometimes nothing really seems <br>&gt; to happen for up to ten seconds, or very little data gets &nbsp;written. Might this be a problem <br>&gt; of the amd64 system having only 4GB of RAM? Any tuneable sysctls? Enabling prefetch didn't help...: <br><br>The most likely cause is that several of your disks are not performing <br>properly. &nbsp;Perhaps a disk is performing error recovery, or there is a <br>bad cable, or maybe even too much chassis vibration. &nbsp;It only takes <br>one pokey disk in a vdev to slow down the whole vdev. &nbsp;Using 'iostat <br>-x 10' while your pool is under continous I/O load may help find the <br>pokey disks. <br><br>Bob <br>-- <br>Bob Friesenhahn <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907390&i=0" target="_top" rel="nofollow">bfriesen@...</a>, <a href="http://www.simplesystems.org/users/bfriesen/" target="_top" rel="nofollow">http://www.simplesystems.org/users/bfriesen/</a><br>GraphicsMagick Maintainer, &nbsp; &nbsp;<a href="http://www.GraphicsMagick.org/" target="_top" rel="nofollow">http://www.GraphicsMagick.org/</a><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907390&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907390&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26907314 2009-12-23T12:56:18Z 2009-12-23T12:56:18Z Barry Pederson On 12/23/09 10:41 AM, Bob Friesenhahn wrote: <div class='shrinkable-quote'><br>&gt; On Wed, 23 Dec 2009, Patrick Proniewski wrote: <br>&gt; <br>&gt;&gt; By the way, I'm also interested in ZFS: is the snapshot technology <br>&gt;&gt; available in ZFS the same as the one available in UFS? <br>&gt; <br>&gt; I don't know anything about snapshots in UFS, but snapshots in ZFS are <br>&gt; certainly remarkably different. ZFS uses copy-on-write (COW) whenever a <br>&gt; data block is updated and snapshot creation simply adds a new reference <br>&gt; to existing blocks. The snapshot is made available as a (usually) hidden <br>&gt; directory (/filesystem/.zfs/snapshot/snapname) which contains the <br>&gt; complete filesystem content at the time the snapshot was taken. In my <br>&gt; experience, ZFS snapshots usually take less than a second to complete. <br>&gt; They are so efficient that some systems have snapshots scheduled to be <br>&gt; taken every five minutes as a defense against user/application error. </div><br>I always liked this quote from this writeup on ZFS: <br><br><a href="http://www.sun.com/bigadmin/features/articles/zfs_part2_ease.jsp" target="_top" rel="nofollow">http://www.sun.com/bigadmin/features/articles/zfs_part2_ease.jsp</a><br><br>&quot;...there's virtually no overhead at all due to the copy-on-write <br>architecture. In fact, sometimes it is faster to take a snapshot rather <br>than free the blocks containing the old data!&quot; <br><br>That's certainly not the case with UFS snapshots, which can take a long <br>time to complete (we're talking freezing your machine's disk activity <br>for many minutes), and are limited to 20 total. <br><br>&nbsp; &nbsp; &nbsp; &nbsp; Barry <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907314&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907314&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26907267 2009-12-23T12:52:03Z 2009-12-23T12:52:03Z Rich-105 I've presently seen some traffic flying around about ZFS performance <br>being sub-disk-saturating on FBSD, and a hack solution of twiddling <br>the ZFS thread priority. <br><br>You might want to look into that. <br><br>- Rich <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907267&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26907267&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26906998 2009-12-23T12:26:12Z 2009-12-23T12:26:12Z Solon Lutz Hi, <br><br>I opted for two 12-disc raidz2. <br>Reasons were: Space is more important than performance. <br><br>But performance is very poor - have a look at the iostats - sometimes nothing really seems <br>to happen for up to ten seconds, or very little data gets &nbsp;written. Might this be a problem <br>of the amd64 system having only 4GB of RAM? Any tuneable sysctls? Enabling prefetch didn't help...: <br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;capacity &nbsp; &nbsp; operations &nbsp; &nbsp;bandwidth &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; capacity &nbsp; &nbsp; operations &nbsp; &nbsp;bandwidth <br>pool &nbsp; &nbsp; &nbsp; &nbsp; used &nbsp;avail &nbsp; read &nbsp;write &nbsp; read &nbsp;write &nbsp; pool &nbsp; &nbsp; &nbsp; &nbsp; used &nbsp;avail &nbsp; read &nbsp;write &nbsp; read &nbsp;write <br>---------- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp; ---------- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- &nbsp;----- <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp;1.23K &nbsp; &nbsp; &nbsp;0 &nbsp; 153M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;510 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;510 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;835 &nbsp; &nbsp; &nbsp;0 &nbsp; 104M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;645 &nbsp; &nbsp; &nbsp;0 &nbsp;68.7M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;474 &nbsp; &nbsp; &nbsp;0 &nbsp;59.3M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;5.49K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;494 &nbsp; &nbsp; &nbsp;0 &nbsp;61.8M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;616G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 53 &nbsp; &nbsp; &nbsp;0 &nbsp;6.73M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;0 &nbsp;39.4K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 30 &nbsp; &nbsp; &nbsp;0 &nbsp; 140K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 33 &nbsp; &nbsp; &nbsp;0 &nbsp;72.8K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 65 &nbsp; &nbsp; &nbsp;0 &nbsp; 162K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 71 &nbsp; &nbsp; &nbsp;0 &nbsp;8.92M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;636 &nbsp; &nbsp; &nbsp;0 &nbsp;79.4M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;10.5K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;485 &nbsp; &nbsp; &nbsp;0 &nbsp;60.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 85 &nbsp; &nbsp; &nbsp;0 &nbsp;10.7M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 15 &nbsp; &nbsp; &nbsp;0 &nbsp;39.9K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 17 &nbsp; &nbsp; &nbsp;0 &nbsp;40.4K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 31 &nbsp; &nbsp; &nbsp;0 &nbsp;73.3K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 62 &nbsp; &nbsp; &nbsp;0 &nbsp; 156K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 17 &nbsp; &nbsp; &nbsp;0 &nbsp;2.24M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;189 &nbsp; &nbsp; &nbsp;0 &nbsp;23.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;9 &nbsp; &nbsp; &nbsp;0 &nbsp;14.0K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; 74 &nbsp; 133K &nbsp; 247K store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 12 &nbsp; &nbsp; &nbsp;0 &nbsp;17.5K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; 255K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;100 &nbsp; &nbsp; &nbsp;0 &nbsp;12.6M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; 1012 &nbsp; &nbsp; &nbsp;0 &nbsp; 126M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 33 &nbsp; &nbsp; &nbsp;0 &nbsp; 137K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;0 &nbsp; 293K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;153 &nbsp; &nbsp; &nbsp;0 &nbsp;19.2M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp;109 &nbsp; &nbsp; &nbsp;0 &nbsp;13.7M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp;15.5K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp;1.54K &nbsp; &nbsp; &nbsp;0 &nbsp; 188M <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;0 &nbsp; 888K &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;556 &nbsp; &nbsp; &nbsp;0 &nbsp;69.5M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp;11.0K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;500 &nbsp; &nbsp; &nbsp;0 &nbsp;62.5M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;0 &nbsp; 766K &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 16 &nbsp; &nbsp; &nbsp;0 &nbsp;30.4K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 29 &nbsp; &nbsp; &nbsp;0 &nbsp; 136K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 38 &nbsp; &nbsp; &nbsp;0 &nbsp;84.7K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 57 &nbsp; &nbsp; &nbsp;0 &nbsp; 145K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 92 &nbsp; &nbsp; &nbsp;0 &nbsp;11.6M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;497 &nbsp; &nbsp; &nbsp;0 &nbsp;62.1M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp;9.48K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;534 &nbsp; &nbsp; &nbsp;0 &nbsp;66.7M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;153 &nbsp; &nbsp; &nbsp;0 &nbsp;19.2M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 12 &nbsp; &nbsp; &nbsp;0 &nbsp;12.0K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 26 &nbsp; &nbsp; &nbsp;0 &nbsp; 101K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 34 &nbsp; &nbsp; &nbsp;0 &nbsp;76.3K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 62 &nbsp; &nbsp; &nbsp;0 &nbsp; 155K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; 75 &nbsp; &nbsp; &nbsp;0 &nbsp;9.48M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp;690 &nbsp; &nbsp; &nbsp;0 &nbsp;86.3M &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;0 &nbsp;12.0K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp;0 &nbsp;14.0K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; 255K <br>backup-7 &nbsp; &nbsp;1.23T &nbsp; 130G &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 store-1 &nbsp; &nbsp; &nbsp;617G &nbsp;15.6T &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 <br><br><br><br><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26906998&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26906998&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26904875 2009-12-23T09:00:05Z 2009-12-23T09:00:05Z Chreo Hi there, <br><br>I've a raidz zpool on 6 1,5TB vdevs running on GELI GEOMs. It has been <br>performing fine until a few days ago when all of a sudden it started to <br>fail being imported. This is on 8-stable <br><br>zpool import reports: <br>zpool import &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <br>&nbsp; pool: Ocean <br>&nbsp; &nbsp; id: 18338821095971722517 <br>&nbsp;state: ONLINE <br>status: One or more devices contains corrupted data. <br>action: The pool can be imported using its name or numeric identifier. <br>&nbsp; &nbsp;see: <a href="http://www.sun.com/msg/ZFS-8000-4J" target="_top" rel="nofollow">http://www.sun.com/msg/ZFS-8000-4J</a><br>config: <br><br>&nbsp; &nbsp; Ocean &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE <br>&nbsp; &nbsp; &nbsp; raidz1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ONLINE <br>&nbsp; &nbsp; &nbsp; &nbsp; label/Disk_1.eli &nbsp;ONLINE <br>&nbsp; &nbsp; &nbsp; &nbsp; label/Disk_2.eli &nbsp;ONLINE <br>&nbsp; &nbsp; &nbsp; &nbsp; label/Disk_3.eli &nbsp;ONLINE <br>&nbsp; &nbsp; &nbsp; &nbsp; label/Disk_4.eli &nbsp;UNAVAIL &nbsp;corrupted data <br>&nbsp; &nbsp; &nbsp; &nbsp; label/Disk_5.eli &nbsp;UNAVAIL &nbsp;corrupted data <br>&nbsp; &nbsp; &nbsp; &nbsp; label/Disk_6.eli &nbsp;ONLINE <br><br>but doing the actual import simply cause the process to halt it's <br>progress (at least a ktrace does not reveal any activity). <br><br>I've also tried doing the import using: <br>zpool import -f -o failmode=continue Ocean <br>with the same result. <br><br>Right, so my first thought was that this has been caused by the <br>notorious uberblock issue (where the . Unfortunately doing <br>zdb -uuv -e Ocean <br>also stops responding (no disk activity noted after 1s of running the <br>command). <br><br>zdb -l /dev/label/Disk_X.eli <br>Reports identical data for all 6 drives (they only differ in the GUID as <br>expected). <br><br>Now running on 7-stable instead (the system was upgraded a monthe ago) <br>does not change the behaviour fundamentally, import still stops <br>responding, however zdb -e Ocean now cause a segfault so it's not much <br>of an &quot;improvment&quot;. <br><br>Any ideas as to how I could get this into a importable state or find out <br>what went backwards? <br><br>Cheers <br>Christian Elmerot <br><br><br><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26904875&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26904875&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26904490 2009-12-23T08:41:34Z 2009-12-23T08:41:34Z Bob Friesenhahn On Wed, 23 Dec 2009, Patrick Proniewski wrote: <br><br>&gt; By the way, I'm also interested in ZFS: is the snapshot technology <br>&gt; available in ZFS the same as the one available in UFS? <br><br>I don't know anything about snapshots in UFS, but snapshots in ZFS are <br>certainly remarkably different. &nbsp;ZFS uses copy-on-write (COW) whenever <br>a data block is updated and snapshot creation simply adds a new <br>reference to existing blocks. &nbsp;The snapshot is made available as a <br>(usually) hidden directory (/filesystem/.zfs/snapshot/snapname) which <br>contains the complete filesystem content at the time the snapshot was <br>taken. &nbsp;In my experience, ZFS snapshots usually take less than a <br>second to complete. &nbsp;They are so efficient that some systems have <br>snapshots scheduled to be taken every five minutes as a defense <br>against user/application error. <br><br>Bob <br>-- <br>Bob Friesenhahn <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26904490&i=0" target="_top" rel="nofollow">bfriesen@...</a>, <a href="http://www.simplesystems.org/users/bfriesen/" target="_top" rel="nofollow">http://www.simplesystems.org/users/bfriesen/</a><br>GraphicsMagick Maintainer, &nbsp; &nbsp;<a href="http://www.GraphicsMagick.org/" target="_top" rel="nofollow">http://www.GraphicsMagick.org/</a><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26904490&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26904490&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26899666 2009-12-23T01:12:44Z 2009-12-23T01:12:44Z Patrick Proniewski Hello, <br><br>I'm playing a little bit with freebsd snapshots (on UFS, freebsd 6.4). &nbsp; <br>And I can't find the answer to one of my questions. <br>If I understand correctly: <br>- at time=0 a snapshot contains nothing but a bit/block map, and every &nbsp; <br>block in the block map is either a pointer to &quot;not used&quot; or &quot;not &nbsp; <br>copied&quot;. <br>The pointer to &quot;not used&quot; is used when the corresponding block on the &nbsp; <br>live file system is empty. The pointer &quot;not copied&quot; is used when the &nbsp; <br>corresponding block on the live FS has not changed since time=0. <br>- at time&gt;0 every non-empty block on the live FS that is to be &nbsp; <br>modified, is first copied in the snapshot, and then, the pending &nbsp; <br>modification is committed. The pointer in the snapshot's block map &nbsp; <br>changes from &quot;not copied&quot; to the address of the copied block in the &nbsp; <br>snapshot. <br>But what about empty blocks? I can't find any information about them. <br>It seems logical to me that empty blocks receiving new data on the &nbsp; <br>live FS will stay as pointers to &quot;not used&quot; in the snapshot, instead &nbsp; <br>of pointing to an empty block that would be copied in the snapshot and &nbsp; <br>grow it's size. I've not found any piece of documentation that clarify &nbsp; <br>this. <br><br>By the way, I'm also interested in ZFS: is the snapshot technology &nbsp; <br>available in ZFS the same as the one available in UFS? <br><br>thanks, <br>patpro <br><br> tag:old.nabble.com,2006:post-26896975 2009-12-22T17:41:52Z 2009-12-22T17:41:52Z Rich-105 <a href="http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763" target="_top" rel="nofollow">http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763</a>&nbsp;may be <br>useful to you - it's what we did when we got stuck in a resilver loop. <br>I recall being in the same state you're in right now at one point, and <br>getting out of it from there. <br><br>I think if you apply that patch, you'll be able to cancel the <br>resilver, and then resilver again with the device you'd like to <br>resilver with. <br><br>- Rich <br><br>On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26896975&i=0" target="_top" rel="nofollow">000.fbsd@...</a>&gt; wrote: <div class='shrinkable-quote'><br>&gt; Steven Schlansker wrote: <br>&gt;&gt; <br>&gt;&gt; As a corollary, you may notice some funky concat business going on. <br>&gt;&gt; This is because I have drives which are very slightly different in size (&lt; <br>&gt;&gt;  1MB) <br>&gt;&gt; and whenever one of them goes down and I bring the pool up, it helpfully <br>&gt;&gt; (?) <br>&gt;&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt;&gt; This is extremely frustrating... is there any way to fix that?  I'm <br>&gt;&gt; eventually going to keep expanding each of my drives one megabyte at a <br>&gt;&gt; time <br>&gt;&gt; using gconcat and space on another drive!  Very frustrating... <br>&gt; <br>&gt; You can avoid it by partitioning the drives to the well known 'minimal' size <br>&gt; (size of smallest disk) and use the partition instead of raw disk. <br>&gt; For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>&gt; of ad12p1 (if you creat partitions by gpart) <br>&gt; <br>&gt; You can also use labels instead of device name. <br>&gt; <br>&gt; Miroslav Lachman <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26896975&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26896975&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; </div><br><br><br>-- <br><br>If you are over 80 years old and accompanied by your parents, we will <br>cash your check. <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26896975&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26896975&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26895970 2009-12-22T15:34:57Z 2009-12-22T15:34:57Z Jeff Roberson-2 On Tue, 22 Dec 2009, Andrew Snow wrote: <br><br>&gt; <br>&gt; Is there any provision to put the journal on a seperate device? &nbsp;This can <br>&gt; solve the performance degradation issue. <br><br>Currently there is no plan for it. &nbsp;The amount of journal data is so small <br>compared to typical journaled filesystems I don't think we'll see a <br>significant slowdown due to the journal writes. <br><br>Consider the operation of creating a new file. &nbsp;We must write a cyl group <br>to allocate a bitmap, an inode block to initialize the new inode, a <br>directory block to add the entry, and potentially the directory inode to <br>update timestamps, directory size, etc. &nbsp;So 4 block size writes if there <br>is no directory allocation. &nbsp;The journal adds one 32byte entry to a block <br>size write that can commonly hold 512 entries if you can aggregate them. <br><br>Another way to look at it is that you can create 512 files (ignoring <br>directory block allocation) before you need to write the journal. <br><br>With full metadata block journaling you're writing the full size of each <br>of those 4 blocks every time a change is made. &nbsp;On a 16k block filesystem <br>that's 64k vs our 32 bytes. <br><br>We can accomplish this because the recovery operation is smart enough to <br>parse all the metadata and see how far along the operation is. <br>Softupdates orders writes well enough that we only have to handle a small <br>number of failure cases and can make big assumptions about the state of <br>the filesystem. &nbsp;If any of these assumptions are not true we still have a <br>full fsck to fall back on. <br><br>Thanks, <br>Jeff <br><br>&gt; <br>&gt; <br>&gt; - Andrew <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895970&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895970&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895970&i=2" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895970&i=3" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26895826 2009-12-22T15:15:44Z 2009-12-22T15:15:44Z Miroslav Lachman Steven Schlansker wrote: <br>&gt; As a corollary, you may notice some funky concat business going on. <br>&gt; This is because I have drives which are very slightly different in size (&lt; &nbsp;1MB) <br>&gt; and whenever one of them goes down and I bring the pool up, it helpfully (?) <br>&gt; expands the pool by a whole megabyte then won't let the drive back in. <br>&gt; This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>&gt; eventually going to keep expanding each of my drives one megabyte at a time <br>&gt; using gconcat and space on another drive! &nbsp;Very frustrating... <br><br>You can avoid it by partitioning the drives to the well known 'minimal' <br>size (size of smallest disk) and use the partition instead of raw disk. <br>For example ad12s1 instead of ad12 (if you creat slices by fdisk) <br>of ad12p1 (if you creat partitions by gpart) <br><br>You can also use labels instead of device name. <br><br>Miroslav Lachman <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895826&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895826&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26895482 2009-12-22T14:18:20Z 2009-12-22T14:18:20Z Steven Schlansker-4 Hello fellow FreeBSDers, <br>I've got a nice shiny ZFS raidz2 array set up, <br>but it's gotten stuck in a DEGRADED state and I can't <br>figure out how to recover! <br>Here's the array as it stands now: <br><br>[steven@universe:~]% sudo zpool status <br>&nbsp; pool: universe <br>&nbsp;state: DEGRADED <br>&nbsp;scrub: scrub in progress for 0h9m, 2.19% done, 6h41m to go <br>config: <br><br>&nbsp; &nbsp; &nbsp; &nbsp; NAME &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; STATE &nbsp; &nbsp; READ WRITE CKSUM <br>&nbsp; &nbsp; &nbsp; &nbsp; universe &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; raidz2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEGRADED &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad16 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; replacing &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;UNAVAIL &nbsp; &nbsp; &nbsp;0 5.93K &nbsp; &nbsp; 0 &nbsp;insufficient replicas <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 3961920099899285277 &nbsp;UNAVAIL &nbsp; &nbsp; &nbsp;0 7.11K &nbsp; &nbsp; 0 &nbsp;was /dev/concat/back0/old <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 6170688083648327969 &nbsp;UNAVAIL &nbsp; &nbsp; &nbsp;0 7.11K &nbsp; &nbsp; 0 &nbsp;was /dev/ad12 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; concat/back2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad10 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; concat/ad4ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ad24 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; concat/ad6ex &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ONLINE &nbsp; &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 &nbsp; &nbsp; 0 <br><br>errors: No known data errors <br><br>One of my drives failed. &nbsp;I replaced it, and in the process of replacing <br>accidentally pulled the other drive. &nbsp;Now, I can't seem to fix it in any way - <br><br>[steven@universe:~]% sudo zpool replace universe 3961920099899285277 ad26 <br>cannot replace 3961920099899285277 with ad26: cannot replace a replacing device <br>[steven@universe:~]% sudo zpool replace universe 6170688083648327969 ad26 <br>cannot replace 6170688083648327969 with ad26: cannot replace a replacing device <br>[steven@universe:~]% sudo zpool detach universe 6170688083648327969 <br>cannot detach 6170688083648327969: no valid replicas <br>[steven@universe:~]% sudo zpool detach universe 3961920099899285277 <br>cannot detach 3961920099899285277: no valid replicas <br><br><br>Any thoughts? <br><br>As a corollary, you may notice some funky concat business going on. <br>This is because I have drives which are very slightly different in size (&lt; 1MB) <br>and whenever one of them goes down and I bring the pool up, it helpfully (?) <br>expands the pool by a whole megabyte then won't let the drive back in. <br>This is extremely frustrating... is there any way to fix that? &nbsp;I'm <br>eventually going to keep expanding each of my drives one megabyte at a time <br>using gconcat and space on another drive! &nbsp;Very frustrating... <br><br>Thank you for any suggestions, <br>Steven_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895482&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26895482&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26893032 2009-12-22T11:28:26Z 2009-12-22T11:28:26Z Bob Friesenhahn On Sun, 20 Dec 2009, Rick Macklem wrote: <br>&gt;&gt; <br>&gt; You could try the patch at: <br>&gt; <a href="http://people.freebsd.org/~rmacklem/patches/freebsd8-clntvc.patch" target="_top" rel="nofollow">http://people.freebsd.org/~rmacklem/patches/freebsd8-clntvc.patch</a><br>&gt; (Fixes an issue w.r.t. client side TCP reconnects and didn't quite make <br>&gt; it into 8.0.) <br>&gt; <br>&gt; I am trying to keep a list of FreeBSD8.0 NFS fixes at: <br>&gt; <a href="http://people.freebsd.org/~rmacklem" target="_top" rel="nofollow">http://people.freebsd.org/~rmacklem</a><br><br>It seems that this patch has eliminated the issues I was seeing. &nbsp;No <br>problems for over a day now, and previously the problem became evident <br>in just a few minutes. <br><br>Thank you very much for your assistance. <br><br>Bob <br>-- <br>Bob Friesenhahn <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26893032&i=0" target="_top" rel="nofollow">bfriesen@...</a>, <a href="http://www.simplesystems.org/users/bfriesen/" target="_top" rel="nofollow">http://www.simplesystems.org/users/bfriesen/</a><br>GraphicsMagick Maintainer, &nbsp; &nbsp;<a href="http://www.GraphicsMagick.org/" target="_top" rel="nofollow">http://www.GraphicsMagick.org/</a><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26893032&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26893032&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26891184 2009-12-22T09:01:32Z 2009-12-22T09:01:32Z linimon Old Synopsis: Kernel panic. msdofs: file name length 266 to large. <br>New Synopsis: [msdosfs] [panic] Kernel panic. msdofs: file name length 266 to large. <br><br>Responsible-Changed-From-To: freebsd-bugs-&gt;freebsd-fs <br>Responsible-Changed-By: linimon <br>Responsible-Changed-When: Tue Dec 22 17:01:08 UTC 2009 <br>Responsible-Changed-Why: <br>Over to maintainer(s). <br><br><a href="http://www.freebsd.org/cgi/query-pr.cgi?pr=141897" target="_top" rel="nofollow">http://www.freebsd.org/cgi/query-pr.cgi?pr=141897</a><br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26891184&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26891184&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26883596 2009-12-21T19:52:38Z 2009-12-21T19:52:38Z Pedro F. Giffuni-4 <br><br><br><br>----- Original Message ---- <br>... <div class='shrinkable-quote'><br>&gt; <br>&gt; Hi, <br>&gt; <br>&gt; Ext3 by default only journals the meta-data <br>&gt; <a href="http://en.wikipedia.org/wiki/Ext3" target="_top" rel="nofollow">http://en.wikipedia.org/wiki/Ext3</a><br>&gt; <br>&gt; From <a href="http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips" target="_top" rel="nofollow">http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips</a><br>&gt; &quot;By default, ext3 partitions mount with the 'ordered' data mode. In <br>&gt; this mode, all data is written to the main filesystem and its metadata <br>&gt; is committed to the journal, whose blocks are logically grouped into <br>&gt; transactions to decrease disk I/O&quot; <br>&gt; <br>&gt; You have to enable full journalling by setting the option in fstab to <br>&gt; journal everything. <br>&gt; <br>&gt; Regards <br>&gt; David N </div><br>I stand corrected. Apparently the data=journal mode is not bad at all <br>though: <br><br><a href="http://www.ibm.com/developerworks/linux/library/l-fs8.html" target="_top" rel="nofollow">http://www.ibm.com/developerworks/linux/library/l-fs8.html</a><br><br>cheers, <br><br>Pedro. <br><br><br>&nbsp; &nbsp; &nbsp; <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883596&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883596&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26883480 2009-12-21T19:28:44Z 2009-12-21T19:28:44Z David N-2 2009/12/22 Pedro F. Giffuni &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883480&i=0" target="_top" rel="nofollow">giffunip@...</a>&gt;: <div class='shrinkable-quote'><br>&gt; ----- Original Message ---- <br>&gt; <br>&gt;&gt; <br>&gt;&gt; On Dec 20, 2009, at 8:37 PM, Pedro F. Giffuni wrote: <br>&gt;&gt; <br>&gt;&gt; &gt; Just wondering... <br>&gt;&gt; &gt; <br>&gt;&gt; &gt; What's wrong with gjournal(8) ? <br>&gt;&gt; &gt; <br>&gt; ... <br>&gt;&gt; <br>&gt;&gt; gjournal(8) journals everything, that is all data and metadata are journaled. <br>&gt;&gt; Which can help with random writes, but essentially cuts linear write throughput <br>&gt;&gt; in half. <br>&gt;&gt; <br>&gt;&gt; Regards, <br>&gt;&gt; Niki Denev <br>&gt; <br>&gt; <br>&gt; I recall ext3fs also journals everything by default and still is very popular. <br>&gt; <br>&gt; I am asking because I've been playing a bit with Aditya's ext2fs (mostly UFS1) <br>&gt; and one of the ideas there is adding gjournal support instead of starting from <br>&gt; scratch. <br>&gt; <br>&gt; Pedro.. <br>&gt; <br>&gt; <br>&gt; <br>&gt; _______________________________________________ <br>&gt; <a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883480&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br>&gt; <a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>&gt; To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883480&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>&gt; </div><br>Hi, <br><br>Ext3 by default only journals the meta-data <br><a href="http://en.wikipedia.org/wiki/Ext3" target="_top" rel="nofollow">http://en.wikipedia.org/wiki/Ext3</a><br><br>&gt;From <a href="http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips" target="_top" rel="nofollow">http://wiki.archlinux.org/index.php/Ext3_Filesystem_Tips</a><br>&quot;By default, ext3 partitions mount with the 'ordered' data mode. In <br>this mode, all data is written to the main filesystem and its metadata <br>is committed to the journal, whose blocks are logically grouped into <br>transactions to decrease disk I/O&quot; <br><br>You have to enable full journalling by setting the option in fstab to <br>journal everything. <br><br>Regards <br>David N <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883480&i=3" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883480&i=4" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26883272 2009-12-21T18:48:30Z 2009-12-21T18:48:30Z Ben Schumacher On Mon, Dec 21, 2009 at 7:09 PM, Andrew Snow &lt;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883272&i=0" target="_top" rel="nofollow">andrew@...</a>&gt; wrote: <br>&gt; Is there any provision to put the journal on a seperate device?  This can <br>&gt; solve the performance degradation issue. <br><br>I'm not sure that would provide much more reliability, however. If <br>you're adding a second device to use the journal you risk losing data <br>if either device fails... given how small the data is (I believe 32 <br>bytes per entry), I'm not sure it's really worthwhile. Most of what <br>I've read in the past about FreeBSD's FS interactions seems to imply <br>that the lower-level (ATA-layer) is not terribly optimized for larger <br>drives and faster interfaces available these days. I'm sure I could go <br>Google up some references, but I think it's been well covered in the <br>past. <br><br>Cheers, <br>Ben <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883272&i=1" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26883272&i=2" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br> tag:old.nabble.com,2006:post-26882998 2009-12-21T18:09:49Z 2009-12-21T18:09:49Z Andrew Snow-2 <br>Is there any provision to put the journal on a seperate device? &nbsp;This <br>can solve the performance degradation issue. <br><br><br>- Andrew <br>_______________________________________________ <br><a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26882998&i=0" target="_top" rel="nofollow">freebsd-fs@...</a> mailing list <br><a href="http://lists.freebsd.org/mailman/listinfo/freebsd-fs" target="_top" rel="nofollow">http://lists.freebsd.org/mailman/listinfo/freebsd-fs</a><br>To unsubscribe, send any mail to &quot;<a href="http://old.nabble.com/user/SendEmail.jtp?type=post&post=26882998&i=1" target="_top" rel="nofollow">freebsd-fs-unsubscribe@...</a>&quot; <br>