Posted on
by

3 Nov 2009
I have recently been conducting a little research into hosting companies/ISPs/data centres to understand more about their speed.

One hosting provider in the UK, UKFast, has recently been marketing the advantages of speed as a prime factor.  Consequently, they have allegedly invested 25% of their profits year on year into improving their internet connectivity while at the same time ensuring that they never exceed [by that I infer “sell more than”] 40% of total bandwidth available*.  Fair play – we all like stuff to be faster.  I was also pointed to a 3rd party web site who provide speed measuring of UK-based web hosting providers – WebCop.
* I was told this by a UKFast sales representative.

I was interested by WebCop’s claims, namely that by distributing their testing servers across UK-based hosting centres, they eliminate bias of one or another datacentre and concentrate instead of the actual, average throughput delivered by them.  It’s a fair claim, but there could be issues.  Today, I sent them this message:

Hi,

I’m interested by your web hosting speed statistics, for two main reasons.

 Firstly, there isn’t much info on your site about how you conduct tests – e.g. which web sites are used to measure the hosting companies relative speed.  This concerns me, as hosting companies can easily make the most prominent web sites the fastest, putting them on the front line of the data centre, while allocating less bandwidth to smaller web sites.

Secondly, you don’t mention from where you test each hosting company’s sites/servers.  So, for example, you could be testing a London-based server using servers in Manchester and Leeds, but the contention in one direction may be significantly higher than in the other direction.  Therefore, you could have skewed results.  In addition to this, if one hosting provider/ISP has a faster network, how can you prove this by testing on their competitors’ slower networks?

I’m looking forward to hearing back from them.  Currently UKFast appears to have leapt ahead in terms of the speed ratings, according to WebCop.

Whois WebCop?

Good question.  I ran a on webcop.co.uk and found that the domain is registered by a company in the Netherlands who has a POBox address in Gibraltar!  Because whois output is subject to Nominet copyright, I cannot redistribute it here.  But if you want to see it, try www.123-reg.co.uk.

I have tried to dig a little deeper; the web is very unrevealing of a company that seemingly wants to stay hidden. I did find out that UKFast’s sister brand, GraphiteRack.com, registered their domain name through ENom, the same registrar that WebCop used, but nothing more.

The public-facing WebCop server seems to be hosted by Tagadab.com, a Clara.net Group company. Interesting that a company (WebCop) with testing servers distributed across the UK, use a London-based ISP with only 6 IP addresses allocated from IANA and some very “comptetitive” prices.  Perhaps they want to keep their web traffic well away from testing servers…

Stay tuned…

 5 Nov 2009
Not heard anything from WebCop yet…

 9 Nov 2009
I got a reply from WebCop:

Our testing servers are located on many different networks and effort has been taken to ensure that they are also geographically evenly located throughout the country. This means that if we test a server located in London it will be tested from all over the country and the average result given. This allows us to display results that are averaged not only across different provider’s networks but also  across different geographical locations.

As for your first point, we are currently addressing this and looking to find the best way to ensure that providers don’t cheat in the same way we know they do for Webperf testing. Currently for the larger providers we test a server located in the standard customer space and not their main website, and for smaller providers we test their company website server. We are looking for a way to make this fairer and are working with some of the larger providers to do this.

On the surface this is a fair strategy. However, it’s very, very easy for a data centre to prioritise traffic to/from a particular machine.  My feeling is that this could be happening already although, of course, I can prove nothing.

My gut instinct tells me that if the majority of datacentres in the UK felt they could increase sales by claiming the fastest network connectivity, they would.

However, every UK datacentre (apart from one) seems to hover around the same speed of connectivity, which suggests that either the system of tests is not respected amongst the datacentre community (in other words, it isn’t perceived as being particularly accurate), or the service provided by one is much faster than the bigger ISPs with which it peers… which seems rather unlikely.

I respect the WebCop team for this endeavour, but strongly feel that until the testing methodology is properly published for the networking and datacentre community, there can be little value in its findings.

Posted on
by

Currently having a couple of issues with laptop and server.   Hmm.. sratching head, thinking cap on, etc..

Problem 1 – laptop swap
On my laptop, I recently resized my root partition (lv) and removed/recreated my swap partition (also a logical volume).  On my new swap, I used mkswap, added an entry in fstab and turned it on.  All rudimentary stuff. 

But when the system came to using it, it hung.  No response in X whatsoever, although there seemed to be disk polling going on, suggesting the kernel was still operational.  I couldn’t flip to another console or SSH in to find out, though.

I created a new partition directly on the disk, not using LVM, and made that a swap too. Same set up procedure as before, then activated it.  This time, when the system needed to swap, it did – as you would expect it to.  Bizarre.  I can’t think why this might be happening, apart from something going wrong through the device mapper, maybe.

Problem 2 – server dump
The second problem I’m having is backing up the server using dump.  In short, when I dumped out a level 0 backup, not all my files were copied.  Strangely, also, directory sizes on the tape, and when restored, seemed padded/boundary-aligned – e.g. 4kb, 8kb or 16kb.  I’m trying to solve this one too, and am using tar in the meantime (which, if testing proves positive, may stick with).

Posted on
by
The full title of this blog should really be ‘SELinux is preventing mysqld (mysqld_t) “search” to ./tmp (public_content_rw_t)’ as that is the problem I’ve been having with CentOS recently (and hence my searches on the web for a solution).

The cause of the problem

I use SugarCRM for customer and project management data – and very good it is too! (Gratuitous plug – I can help your company install and use this fine software :-) ). Except that recently, when listing my Accounts within Sugar, I would not see all of the account context. Only the account data itself would be displayed and none of the subpanels/links.

The query to retrieve more data was failing, with this error message displayed in the browser window:
mysqld: Can't create/write to file '/tmp/#08y2jw' (Errcode: 13)

In my system log (/var/log/messages), I also got multiple SELinux errors like this:
Oct 13 09:07:50 server setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./tmp (public_content_rw_t). For complete SELinux messages. run sealert -l 1762c478-f3a2-4eeb-be09-bd3dc037d945

Clearly, the reason for “Errcode: 13″ was due to SELinux.

Incidentally. if you have seen a similar error on your web site, but with (Errcode: 28) instead, this is likely due to shortage of disk space. A great way of determining operating system errors like this, is to use ‘PError’, thus:
# perror 28
OS error code 28: No space left on device
# perror 13
OS error code 13: Permission denied

So there we are – two distinct and different issues.

With SELinux, resolving the permission issue can be difficult. By issuing # sealert -l 1762c478-f3a2-4eeb-be09-bd3dc037d945, as suggested above, I got the following output (trimmed and highlighted for clarity):

Summary:
SELinux is preventing mysqld (mysqld_t) “search” to ./tmp (public_content_rw_t).
Allowing Access:
Sometimes labeling problems can cause SELinux denials. You could try to restore
the default system file context for ./tmp,
restorecon -v ‘./tmp’
Additional Information:
Source Context root:system_r:mysqld_t
Target Context system_u:object_r:public_content_rw_t

First things first: issuing # restorecon -v './tmp' didn’t fix it for me. I was also surprised to see that the path to /tmp was relative to the current working directory, so I tried a slightly modified # restorecon -v '/tmp', but to no avail. After restarting mysqld, the problem persisted: MySQL was simply being refused access to /tmp. Somewhere, a policy is disallowing this.

It’s a mistake to assume the the source context and target context should be the same; they don’t have to be, as it’s entirely policy-driven.  I made bold those aspects (the file Type) above to highlight this incorrect assumption (that I previously held).

Find and fix a policy?

Although finding the troublesome policy and analysing it is a Good Thing, it’s also time-consuming and requires significant knowledge of SELinux, chiefly to avoid creating security holes. A better way, I found, was simply to relocate where mysqld tries to store temporary data.

Thanks to Surachart Opun’s blog, I learned that you can specify a new location for temporary files. In /etc/my.cnf, add or edit the following:

[mysqld]
tmpdir=/tmp # # e.g.
tmpdir=/var/lib/mysql/tmp

Now do the legwork to set up the directory properly:

First, create directory with appropriate permissions
# cd /var/lib/mysql
# mkdir tmp
# chown mysql:mysql tmp
# chmod 1750 tmp

Now set the SELinux context up:
# chcon --reference /var/lib/mysql tmp

and make the SELinuiux context permanent:
# semanage fcontext -a -t mysql_db_t "/var/lib/mysql/tmp(/.*)?"

Finally, restart mysql:
# service mysqld restart

Closing thoughts: optimisation

The methods above fixed the particular problem I was having. They didn’t, however, actually pinpoint the cause. This is one of the good things about Linux and SELinux in particular: you are forced to rethink what the system is doing and work out a solution that sits within the predefined security context – or learn how to write SELinux policies. Personally, I prefer the former ;-)

There is an additional benefit to the solution above – namely, optimisation. Because we have specified the security context with semanage, we are free to mount an external file system and use that instead for MySQL’s temporary files. In other words, we can maintain the security but increase the performance.  One such filesystem could be tmpfs. tmpfs is actually a RAM Disk, uses a fixed amount of RAM to provide file storage. It is much quicker than an on-disk filesystem and thus perfectly optimised for storing temporary, caching data. There are many resources about tmpfs on the web. A good introduction to tmpfs can be at Planet Admon.

Posted on
by

I recently found myself having the need to revoke an old certificate. The steps are actually quite straightforward, but you do need to have your old revocation certificate to hand.

For more info, visit the GNU Privacy Guard site: http://www.gnupg.org/gph/en/manual.html

Simple follow these steps. In a terminal, issue:

  • gpg –import my-old-key@mydomain.com (0x712AC328) rev.asc
  • gpg –keyserver certserver.pgp.com –send-key 712AC328

That’s it!

Posted on
by

My eyes have been opened to yet another case of foul play by a megacorp.

As if their package management isn’t disgusting enough
(www.theregister.co.uk/2008/07/23/enormouse/), /their BIOS
configuration on laptops of the recent few years leaves MUCH to be desired.

On the bottom of my HP Compaq 6715b laptop is a removable panel which
covers a memory slot and a Mini-PCI-E connector. “Great”, I thought,
having a non-functioning Broadcom card in there, “I’m going to install
an Intel 4925 AGN wireless card here because it’s supported by the
firmware/kernel I use (CentOS 5.3) – and I’m loath to building a new
kernel when I can just plug in a new card ;-)”.

My card, £15 off Ebay, arrived this morning and I carefully fitted it.
Booted the machine, went into the BIOS settings to ensure it was
enabled, and…. wait a minute, it isn’t listed. Perhaps it’s broken…
or …perhaps HP have imposed a blacklist of vendors/subsystems which
THEY don’t allow to be recognised in MY computer. Not listed in lspci,
nor dmesg… basically nowhere.

Is this legal? Did I ever see any restriction declared ANYWHERE before
buying this machine that stated “HP retains sole right to how this
machine is used and with what”..?

What point is there putting this restriction in?! Someone buying a
budget laptop isn’t going to source their over-priced parts from the
OEM! Why, darn it, why?!

If you have – or rather, are thinking of buying an HP, Dell or IBM
laptop, I’d suggest reading these first:
– http://www.engadget.com/2005/02/22/the-hp-bios-that-locks-non-whitelisted-mini-pci-cards-out/
– http://www.aigarius.com/blog/2008/02/07/sneaky-blacklisting-of-wifi-in-hp-laptops/
–
– http://www.richud.com/HP-Pavilion-104-Bios-Fix/
– http://www.paul.sladen.org/thinkpad-r31/wifi-card-pci-ids.html

I’ve thought about actually re-flashing my BIOS with modified code,
partly out of sheer bloody-mindedness towards HP (oh, and I would
publish, intricately, the solution), and partly just out of the
practical need for wireless networking. But now, I’m just baffled by
the whole thing.

Hilariously, as a final insult, the latest BIOS update from HP for my
machine, “updates the Computrace OPTION ROM to version 866”. So…
you’re telling me I have this “Computrace OPTION ROM” installed, huh?

HP – http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=321957&prodSeriesId=3368540&swItem=ob-67085-1&prodNameId=3356624&swEnvOID=2094&swLang=8&taskId=135&mode=4&idx=3

BlackHat (deactivate the rootkit) – http://www.blackhat.com/presentations/bh-usa-09/ORTEGA/BHUSA09-Ortega-DeactivateRootkit-PAPER.pdf

Is there no goodness left in the world? Can old ladies not be helped
across the road any more? MUST we buy battery hen eggs instead of free
range?! Like, where’s the love, man..?

Posted on
by

Yes, it seems too good to be true.

Well, guess what?! It IS!!

That’s right. Your old tat (or, you could say, my old tat) is just about as worthless to everyone else as it is to me. I’ve spent ages on ebay and sold almost nothing. And what I have sold, I sold for 99p.

Give to charity instead, that’s what I should have done! Pah!

🙂

Posted on
by

So… sometimes I wonder, what’s the point? I add a bookmark, and the Firefox plug-in asks me if I want to add it to delicious, so generally I say yes as I think it’s probably useful to others. But, syncing with delicious doesn’t seem to retain the organisation I give to my bookmarks (they are my bookmarks, after all).

I have installed xMarks to Firefox, which is great. Delicious, on the other hand… it’s not Digg… what is it?