Starting to diagnose DFS issues

One of my client's server pairs seems to be intermittent in its syncronisation of data. Here are a few commands to test the sync. 

Using an elevated command prompt ( right click, run as Administrator ) execute the following:

C:\Users\ryv> dfsrdiag.exe backlog /SendingMember:ServerX /ReceivingMember:ServerZ /RGName:RepGroup /RFName:"Data"

So what does this do? The initial part of the command is dfsrdiag.exe - the program to run the diagnosis. The switches are as follows:

/SendingMember:ServerX - this is the server pushing data to the replication partner

/ReceivingMember:ServerZ - and this is the server receiving the data

/RGName:RepGroup - DFS Management has servers in groups so it will want the group name

/RFName:"Data" - DFS is capable of managing different folder groups with different sync settings so you have to specify the folder. 

If all is going well the result from this command should come back with something like: 

No backlog - member <ServerZ> is in sync with partner <ServerX>

Operation succeeded.

If it doesn't come back with this then the fun begins. Once I have a good example I'll post that. 

Ubiquity UAP-Outdoor+ Review

Recently for a motel client I rolled out 12 of these little babies. The UAP-Outdoor+ is a compact, apparently weather resistant access point. The website details are here: https://www.ubnt.com/unifi/unifi-ap-outdoor/

I chose these for a number of factors:

• they are designed for the outdoors
• 183M range (apparently)
• 300Mbps speeds
• 802.11b/g/n with plenty of bandwidth and a wide frequency band
• price point was pretty good (compared to other products)

While not strictly producing a meshed network, the UAP-Outdoor+'s have a zero config handoff, so end users won't know as they move around through network range. 

Configuration of the access points was really quite easy - Ubiquity has a great bit of software called UniFi v4.7.6 Controller for Windows. Installing this on my laptop, and connecting it to the same network as the first access point got things up and running really fast. UniFi Controller allows you to configure SSID's, guest network information and everything else about the access points. Love it! Great bit of kit and it made the work of configuring all 12 access points very quick. Ongoing management is excellent too - it shows you information about the clients connected, the data they're using, the most heavily used access points - all sorts of stuff. It also allows you to play with the network topology. In this particular instance, three of the access points are not cabled in - they are "uplinked" via other WAPs. As testing went on, it appeared that my original plan to link certain APs together wasn't as good as the actual data suggested. So I modified it and moved the wirelessly linked access points to other master WAPs. Really excellent and I was very pleased with the rollout. All in all, the budgeted hours for the install were far over the actual hours - always good for the customer and the bottom line.

The range exceeded our expectations significantly and allowed us to push the coverage out to other parts of the property that were designated for a later stage. Needless to say, people were quite happy about the whole thing :-)

I look forward to using the Ubiquity gear again in the future. As for now, I can happily recommend the UAP-Outdoor+ access points!

OTRS 5 Review and Thoughts

Recently I upgraded our several years old OTRS install to version 5. The upgrade procedure is exactly like all the others have been. It was straightforward and I actually found that the documentation was better this time around than it has been in the past. After backing up, upgrading and testing, I like, mostly, what I see. There are a couple of things that are a bit annoying though.

When closing a ticket, the previous version would automatically put in subject in the closure note. Likewise, in the Owner Update note it doesn't do it either. So each time, I have to update it. Not a problem if you're only getting a few notes every now and then, but we are changing and closing tickets in some large numbers at times.

Also, I have about 50 statistics generated monthly via a script I run. The syntax for doing this.

From: /opt/otrs/bin/otrs.GenerateStats.pl -n number -blah blah etc

To: /opt/otrs/bin/otrs.Console.pl Maint::Stats::Generate --number number etc

(from: https://otrs.github.io/doc/manual/admin/stable/en/html/statistics-module.html#stats-managing-the-module)

I like the speed of the updated system and the cosmetic changes are nice too. The menus have changed slightly and they are more dynamic with drop downs. The layout is a bit nicer too but generally the speed of the system is better.

Upgrade to version 5 if you have the chance - seems to be a winner!

The struggle towards productivity

Anyone working in the IT sector knows how hard it is to get a flow going without interruptions. Working in a small business and supplying support for everything in IT to our clients, the phone rings constantly (desk and mobile) and email never stops pinging in the background. Add to this our obsession with multiple screens, a billion web pages open, Twitter, FaceBook, Instagram etc and there is a huge amount of data overwhelming us.

We also get jobs logged from our other staff members, clients can create their own and the phone calls / emails generally generate work to be done. Keeping up with all of this is hard, and the hardest part is keeping all the tasks straight and on point. We use multiple systems - OTRS and another, non-cool job management system for our tasks. This is a nightmare too - jobs coming in to both, keeping them updated and trying to remember if you've actually updated the job or not. Plus trying to keep on track with a complex task long enough to get the thing done.

I've been working with my guys on being task oriented and not worrying too much about time. If they are working on a task for a client, and it's all billable, then move onto the next task and do it, then their time is being used productively and everyone is happy. This seems to be the best way to work with tasks.

I have also found that when the pressure is on, I leave my dual monitor setup and take my laptop off somewhere to work, usually the boardroom with no direct extension and leave the mobile phone on my desk. The more I do this the more I wonder if having dual monitors is a hindrance and paying attention to my multiple mobile devices is worth doing. I read research that suggested after a break in concentration it could take up to 20 minutes to get the focus back on what you were doing.

We have been aiming to have 90 min stretches working on a single task, then a break for a bit, then back into it. It's incredibly hard, but getting the other staff around us on board is making it easier. With a bit of luck we can continue to push this forward. Leave a message in the comments if you have productivity tips that work for you!

XenServer physical moves - traps for new (and old) players

Recently one of our clients relocated their main office and we moved their servers for them. Upon arrival, racking and cabling the XenServer hosts all powered up immediately. The HP StoreEasy NAS with all the data on it also started up.

We noticed that our XenServer hosts had no IP addresses - no data was showing and they seemed to have lost their configuration. Uh oh.... we had a huge bet on getting this thing going by the end of the day with the managing director. There were beers involved and we were pumped to make it work.

After fiddling around with the XenServers and trying to get it working and swearing a *lot*, I turned them all off then back on. Lo and behold they came up and it took a bit to figure out what had gone wrong.

The answer was simple - the XenServer hosts are big, fast machines with a lean install on a high speed SAS drive and the NAS is slower - running Windows Storage Server 2012 and RAID'ed disks. The hosts had come up first, looked for the storage repository and then failed when they couldn't find it. Odd given that I thought the hosts retained most of their settings.

So in future - NAS up first, then start the hosts and all will be happy. The rest of the move was spectacularly successful and we had plenty of beers to drink as a result!

OTRS 5 Review

I've been using OTRS for about 10 years now, starting when I was doing desktop / server support at uni. Since then it's changed a lot in the way it looks, but fundamentally it has remained the same. The great things about OTRS are:

• creating / modifying / updating and closing tickets is easy
• the interface is relatively straightforward
• creation of tickets from emails is easy
• reporting is straightforward
• open source and robust

Since the early versions it's ticked all these boxes and I was very interested to see what OTRS 5 was going to bring. The interface is still the same, updated here and there with a prettier graph showing closed / opened tickets but generally the same. Fonts are still nice and readable and its good for what it does.

We use it for our clients that have a maintenance agreement with us. We have them create tickets that we then update and put minutes against. OTRS has never had a parts component and we have always used a secondary system for that. It does a good job too of keeping tickets together - I can't recommend using the master/slave system enough. Merging tickets has always worked flawlessly too.

The upgrade process is well documented and relatively straightforward. Just be aware of what user you are accessing the system as when you're doing it - sometimes you need to be root and other times it's important to be the OTRS user. All in all, it took about 45 minutes with backups, testing and checking before I let the boys go wild on it.

So while the interface is prettier there are a couple of things that annoy me right off the bat. When I add a note I now have to put something in the title - before it would simply put "Note" in there. Likewise for closing a ticket or changing ownership - this one extra step is annoying because I do it so frequently. Also, we have queues that are underneath a top level queue and that needs to be manually expanded out each time now instead of it sitting out like it always has. Minor annoyances it's true, but annoyances nonetheless.

I suspect most of the changes in the system are in the back end and I'm noticing that it is running more smoothly. We are using a fairly archaic IBM server to host it on an Ubuntu Linux Server platform with about 10K tickets give or take. MySQL is our database of choice and the backups run at around 1 GB in size. Thus far it has all been good. I'm looking forward to seeing how it travels over the next few weeks. I have noticed that the iPhone package is no longer supported - apparently the interface scales with the mobile platform - we will test this further. 

Enjoy :-)

Plone - the open source Content Management System - a review

One of my clients, a non-profit, has a lot of files on it's clients. They need a way to digitally store these files, securely and with availability for certain people. They also need these files to expire and be deleted after a given length of time - usually about 7 years.

These were the parameters I was given to search for a Document Management System (DMS) or more commonly a Content Management System (CMS). There are quite a lot of them, but most are designed for front facing information delivery - that is, to write something, put it up for review, have it reviewed and then published. We do not want this data published ever - and some CMS's make that a bit tricky to manage. So at the end of the day, I looked into several CMS systems that looked like they could be useful.

The first one to be reviewed was OpenKM (www.openkm.com). It looked OK, was open source which is preferable and seemed to have solid security and publishing options. Backing up the database and upgrading OpenKM seemed like a nightmare. I tried a couple of times only to experience great sadness and eventually throwing my hands up in defeat. Scratch that guy off the list.

The next one I seriously considered was Plone (www.plone.org). An open, attractive looking interface certainly helped matters from the get go. I liked it, it looked nice and the decision makers at my client would like it too. Open Source, Linux install - OK good to go. The installation documentation is pretty good, a bit fuzzy at times ( though I do attribute that in part to a lack of solid caffeine) and it was relatively straightforward to get going.

I followed the instructions and started to build the site. It was pretty straightforward and offers different workflow types - so for an internal site, there is no publishing option available. Posts (or uploads of files) are set to either "For Review" or "Private" which is perfect for the application we have at hand here. File size uploads are relatively straightforward to change - there is a config file, much like MediaWiki or similar, that allows for changes to the site that are outside the scope of the in-application setup.

After the build, and setting up internal DNS, outgoing mail and accounts, I set it loose with the client. They have logged in, tested it and come back with a plethora of questions - many many of them. I have changed the security options several times and it's handled it all quite well. Before the system goes into production I'll be making sure backups and restores work. Phone has internal ones that I still need to do more testing on. Stay tuned!

Windows 10 - Upgrading Experiences and general first impressions

I've taken the plunge and upgraded my beloved Pavilion dv7 to Windows 10 from 7. I watched with baited breath yesterday to see if anything would happen, after all, I had signed up to get the upgrade and was prepped for it to happen. Nothing happened. I experienced sadness and then I watched my download widget in Rainmeter jump up to 1.5MB/s - our full download speed here. It stayed there for some time and then trailed off. OMG! I thought exultantly that my laptop would then upgrade magically and instantly.... but alas no. Nothing happened again. Anti-climactic was the theme for Windows 10 day. After several hours where I pointedly ignored my laptop and hammered away on the desktop, I patiently waited for something to happen. Around 4pm it did - a new windows popped up and said it was ready. Just a few things to do over 10 seconds, then the install would start. I click on the Go button and waited. It turns out that 10 seconds lasted 14 hours - I went home, slept, came back and the laptop was sitting in the same spot. I killed the upgrade and then a Windows Update "Updating" popped up. The upgrade started again! Oh the excitement! I pointed it out to my fellow IT chaps here in the office. Their response was underwhelming to say the least.... especially as it seemed to stall at 15% - always a bad sign. Thankfully it got going again and my laptop continued to upgrade. Over the course of an hour it completed and now I sit here, typing away in Edge on Google's Blogger. Apart from layout changes and the GUI interface upgrades, it seems a lot like business as usual. The upgrade brought across all my apps, including RainMeter and all the widgets, Chrome and Firefox and all that other stuff as well. I'll play with it over the course of the day and see how it goes. This is a production laptop and I intend to show it off to people so they know what to expect when the upgrade comes through. First up I quite like the new icons and the look of it. The network stuff all looks very similar - there is quite a blend of the Windows 7 functionality and a bit nicer than windows 8 GUI stuff going on. I don't know that I'm loving the Start Menu - perhaps it will grow on me in time. The ads to "Get Office" annoy me - I already have Office 2013! I did have some issues in the preview "Insider" version where scrolling in the Store menu caused it to crash. That seems to have gone away for the moment. I'll keep at it for a while and see how it goes!

Windows 10 trial impressions

As all the world (not literally) knows, the newest version of the Windows Desktop operating system Windows 10 (X?) arrives at the end of the month. I've been playing with it since it's early days and throughout the updates and newer iterations of it my feelings haven't changed much.

I think the overall vision for the UI is missing internal design cohesion. My initial feeling was that they've tried to take elements of the Windows 8 menus, and blend them into the Windows 7 (and previous) familiar menu style. I don't like it. It's gaudy and unpleasant and for straight business work it doesn't speed things up at all.

My other gripe with it has been that under a VMware Player install, it only starts up once every 3 or 4 times. That concerns me. What also concerns me is the dumbing down of the control panel. Accessing the stuff in there isn't for everyone, but the people who do want to get into it need access to the internals of the machine. I can't help but compare it to Mac OS X. The new release of that 10.10.4 is out, and although there aren't huge differences, I've been able to migrate my MacBook Air from 10.8 through to 10.10.4 without any difficulties, or needing to learn a whole new GUI.

My final gripe, and I'll keep this bitch session relatively short, is the move towards subscription model licensing for Windows and Office. I'm happy to pay for an OS, or a piece of software. No worries. I'm not happy to have to pay every year for it. Particularly when I use about 1% of what Office is capable of doing *and* I can get all I want out of Google Apps, plus my email and blog and website etc for $5 / user / month! It's not that I don't want to reward the people who've worked so hard on this, but frankly, they get bugger all of the money I pay anyway. If I'm being forced to use some tarted up version of Windows, it had better be good. Windows 8 was rubbish, and we've successfully avoided upgrading most of our clients and home machines. The way Windows 10 forces itself to be know with a "free" upgrade worries me. We all know how wonderful the first iteration or two of an OS are, and I can only begin to imagine the problems we'll have with Windows 10 when it comes out.

By the end of the month I guess we'll know more!

Xenserver 6.2 to 6.5 upgrade notes

With 6.5 released we have started upgrading machines. Our initial upgrades were fairly straightforward and although the install of the Xentools was very slow and required numerous restarts it completed fairly well.

We have had issues with 2012 server running Exchange 2013 - insofar that we've lost the network adapter and can't get it back. The solution seems to have been to clone the VM and then remove and add the adaptor back in. The restart took a very long time and there were issues with the Xentools install not working properly and difficulties with getting it to upgrade. We ended up removing it and reinstalling it however that wasn't really optimal and we blew our outage window but quite a margin.

Before upgrading Xentools definitely take a snapshot of the VM if it's running Exchange. As we do more I will continue to update this site with more information.

Things to remember - Google Sheets - how to show the last number in a column

I have a Google Sheets file that I put all the water information for our farm into. It's fairly comprehensive with formulas etc to let me know the following once I measure the water level from the top of the tanks:

• volume of water remaining
• amount used since the last recording date
• average usage since last recording date
• amount of rain we can have until the main collection tank overflows (so I know when to pump it up to the feeder tanks)

I would like a summary of the actual levels in each tank in a nice little chart that I can import into our personal intranet site hosted with Google Sites (this part is ridiculously easy), but I don't want to have to change the site information overtime I think about it. 

I found this:

=FILTER( A10:A100 , ROW(A10:A100) =MAX( FILTER( ArrayFormula(ROW(A10:A100)) , NOT(ISBLANK(A10:A100)))))

Now if you put that into a cell, alter the ranges to suit, it will give you the last number in the column (and maybe row, but I haven't tested that). It works very well - not only for numbers but for the dates as well. The little summary chart now shows:

• The Date of last reading
• amount of water remaining in each tank
• amount of rain we can store until the collection tank overflows

And with Google Sheets adding in some decent conditional formatting I have it all colour coded and it looks a bit like this:

Not bad eh? The bit that tells me the overflow is elsewhere on the page. And yes, we're in pretty decent shape given that it's the start of Autumn and we should get rain soon!

Service - what does it mean to you?

Recently our little company has been very busy and while it has been tricky to maintain a high level of service, it's something we are managing very carefully. Interestingly though, we have had clients ask us to do things quickly, easily or cheaply. Generally there is no difference in the service provided, however, as our time becomes more precious, it's easy to take the client's request and act on it in the manner they have requested.

For example - "Can you quickly wipe these computers and then roll them out to be sold?"

"Sure - how much time and effort do you want us to put in to this?" (read: how much money do you want to spend on using our valuable time).

"Not much, just make sure our data's wiped and then get them out the door."

OK, so we clear the data, wipe the free space or re-install after a Darik's Boot And Nuke (DBAN) and then roll them out.

Now our client says to us that it hasn't been done properly and they want an explanation. For the first time ever, I reckon, I pointed at the email trail and said to them "You wanted fast, easy and cheap. You got it. If you want well done, comprehensive and completely satisfying, that does not fall into the fast/easy/cheap categories".

Interestingly the client sat back for a moment, said "You've got a point there. Can you sort this out properly?" Well sure I can and I have, but the cost in my time is there.

There's a line in the movie Tango and Cash where Jack Palance replies to two of his underlings telling them "Quick and Easy. Quick and Easy is how you make a cake." and then goes on to talk about killing Tango and Cash. My point to the client today was "Quick and Easy is how you make a cake - not have a well thought, solid and reliable IT infrastructure." Wake up peeps - spend some money and do it right the first time and stop being disappointed when you pay minimal dollar and get minimal effort.

XenServer 6.2 update woes

Recently I was required to update a number of XenServers that I've taken over. They were running 6.2SP1 but were about 6 updates behind. No problem, I'll just queue the updates up and shoot them over. Had 2 Windows VM's (2008R2 and 2012 server) and 5 Ubuntu 12.04LTS Linux Servers. The updates were fairly straight forward - I followed my own guide :-)

After the final update was completed I booted up the VMs only to find the two Windows Servers came up no problems, and none of the Linux servers booted. None of them. They started up, black screen and white cursor in the top left hand corner and that was it. No boot no errors, just gone.

I ran through some basic repair work and had no joy. I changed the GrubConf.py from if arg.strip() == "${saved_entry}": to if arg.strip() == "${saved_entry}" or arg.strip() == "${next_entry}":. Still no luck. I re-installed Grub - no joy. I booted from the Ubuntu server CD and tried to run rescue the broken install. On several of the VM's I couldn't even get the system to run a shell in the / file system (/dev/xvda1 typically on these setups). It was incredibly frustrating and after three weeks of banging my head against this, I've still not managed to get this all going.

Luckily I had backups of most of the servers and so I haven't lost everything but I have lost some critical data. The message here, dear friends, is multiple:

• test the updates one at a time to make sure everything starts up. Yes it will take longer and it's a bit more frustrating, but believe me, you want to know if things are going to shit themselves.
• backups backups backups - are not important. RESTORES! are the important part. Believe me, I cried into my beer when I couldn't get one of these machines back up and going - it just wasn't going to happen. I lost a wiki with data that wasn't anywhere else and that's caused me no end of grief. The extra work it's caused is phenomenal and I feel very bad about it indeed.

I've reached out to the collective wisdom of SAGE-AU (www.sage-au.org.au) but sadly no luck yet. I'm open to suggestions!

elementary OS Review

I came across elementary OS on LifeHacker I think and thought it looked pretty interesting. I've been looking for a lightweight operating system that's reliable and fairly full featured (I accept it won't be totally featured - that's the cost of lightweight) for use on various older laptops and the like.

elementary OS is a free download with donations as optional. I downloaded it and burned it to a CD. Alas my favourite test laptop has gone the way of the dodo and is never to return (much like the dodo). I found a HP Pavilion D6 floating around, replaced the hard disk (it was toast) and the RAM (also toast) with 6GB. It's an i3 which doesn't really qualify as a lower end machine, but what the hey.

elementary is based on Ubuntu linux and is quite heavily customised running a lightweight desktop called Pantheon. The current version of elementary is called Luna and it's nice to see they are in development of new versions and it seems like an active development environment. Midori is available for Internet access - I installed Chromium as that's my preferred web browser. There is a calendar app, Geary Mail (which I haven't used), Shotwell for photo organisation and Empathy to tie in with Jabber, Facebook etc.

The interface is slick and looks good and the laptop boots a rocket. All the apps are fast and updates etc are easy as pie. There is an application for updates which is simple and straightforward and the whole thing is well organised and designed. It's quite a joy to use actually. The application install package - Software Center - looks good and is straightforward to use, offering a large range of packages. All the stuff I like to use I was able to install and get on with the job. I have a pretty small requirement list though, in the interests of full disclosure. Chromium has to work, I have to be able to burn CDs and DVD's and I need terminal to be fully functional. Pretty, fast and stable are the other three musts for any OS and I find elementary to check all those boxed.

I now run Mint and elementary side by side on laptops and desktops - the speed difference is quite noticeably between the two, and with their ultra stable Ubuntu (Debian) ancestry it's lovely to use. I've enjoyed using elementary OS Luna and I'll be keeping it on my Pavilion for the future - well as long as the Pavilion holds together!