LDAP and Mysql

January 20th, 2006

Full mysql replication with 5.0 code base is now up and working. Openldap replication with the new 2.3 code base is up and working. Both are replicating over native ssl encrypted connections so we have full redundancy on our information. I am looking to do the cyrus back end short as the new 2.3 code base for cyrus-imapd now includes replication, so this will mean that even if we lose one branch of the network that we will be able to check mail at another server instead of having to wait for the main mail server to come back up (we never lose email when something goes down, but we do have a single point of failure for VIEWING emails). Finally I’m about to seriously change the usage profiles of a number of the machines, this will enable a lot more redundancy to be put in place and to actually start leveraging our distributed nature.

Finally if anyone has a spare machine that is on a reasonable connection, permanently on and that they wouldn’t mind me having root access on, I would appreciate being able to run a couple of more services, things like mail and dns. Let me know.

New additions

September 28th, 2005

While I might have been lazy with regular postings here I have been working on the infrastructure quite a bit with a number of changes. Part of those changes will involve me creating a members only area shortly in which I will outline the infrastructure we now have in place and the directions that I would like to see it going in. On a different note we have two new Disciplina networks sites today. Check out economics and politics if those areas hold an interest to you.

Downtime

September 14th, 2005

Just a quick note to apologise to those who were adversely effected by the recent outages. Terra, one of our main computers started mysteriously going offline over the last week culminating in a blown power adapter Monday afternoon. I have bought and installed a new one which seems to have fixed the problem and taken the opportunity to redo some of the dsnet infrastructure. I will try and get some new diagrams out soon for those that are interested but there will probably a couple of 1-10 minute outages over the next 2 days before every thing is bedded in, at that point I am hoping its back to our usual once a year downtime :)

TOR

June 10th, 2005

Well it looks like I have found something that Disciplina networks can get behind. TOR. Its funny that given how interested in the subject I had no idea about TOR until yesterday. I used to trawl the net looking for things related to anonymity and security but I had given up for some years now, even to the extent of changing the purpose of dsnet itself. Now my interest is rekindled, I am going to see what I can do for TOR. I suggest you check it out here. Let me know what you think.

another blog

May 24th, 2005

Well as if I don’t have enough blogs already I just got a new one at Gentoo. You can check it out here. I was going to let them aggregate from one of my other blogs but then thought better of it. At least for now.

Gentoo Developer

May 19th, 2005

Well its happened. Check out my blog entry at my personal blog.

Websites

May 4th, 2005

I need feedback on the following problem.

Good network design says that each server runs as few services as possible. Less services = less attack vectors = more secure and centralised administration. The problem is that most of the websites that I have setup have sensitive information and should be encrypted, in addition, until I get pubcookies up and running its important that whenever a member enters a password it is done over SSL. Given all of that I need a way to make all our websites ssl encrypted on the one box. The problem is that I have limited IPs so my options are these:
* use vhosts (eg bookshelf.disciplina.net, nagios.disciplina.net) but put the sites on non standard ports (eg https://bookshelf.disciplina.net:444 https://nagios.disciplina.net:445).
* drop the idea of using vhosts and just do something like: https://services.disciplina.net/bookshelf https://services.disciplina.net/nagios
* run websites on more then just one box so that I can use each addition machines IP for a ssl cert.

I personally don’t like the first option, its ugly. The second option I also think is ugly but I am leaning towards it right now. The last option is the best in terms website presentation but bad for security.

What are your thoughts? Let me know what you think I should do.

Monitoring vim bashing lazarus

May 4th, 2005

Well I am doing a number of things right now, in fact too many things to seriously focus on any one and I am going to have to invoke a bit of discipline (pardon the pun) and just start knocking them off. Firstly I have spent a lot of time and got my .bashrc .bash_profile and .vimrc in order. The result is actually really huge, both in terms of effect it has had on my productivity and in actual line numbers. Basically my bash environment is now a lot more aware of gentoo extensions, something I needed to do due to trying to become a gentoo dev, its also a lot more flexible in that it now checks my environment quite extensively before changing things based on whether or not they are present. This enables me to take the bash environment with me to any unix, not just Linux and be confident that it will work and that they will all be usably similar. While I was bashing I had a look at vim. I grew up using vim and I really love it, but I have always been told that emacs is the real solution for hardcore editing. The last few years has seen me to a lot more coding then ever before and the need to have a real editting environment has grown and last month I decided that I needed to do something about it. I thought about learning emacs briefly but discarded the idea almost immediately due to having to relearn so much, so instead focussed on vim which I knew there was a lot more to it then I knew. After sitting down and working through some other peoples extensive .vimrc’s I am happy to say I now have a really nice IDE style environment for vim! All I can say to you part time vim users out there is take the time to explore it further, its worth it. All of these files will be up on a published version control system of some kind soon, the main hold up being I am not sure if I should just go ahead and implement subversion or wait and see how well git works out. Suggestions as to what VCS to use are welcome, but bear in mind DS net does NOT really need a distributed model like git / arch / mono / bk, though thats not to say that it shouldn’t either.
On an amusing side note, I took some ram OUT of aqua and booted it and lo and behold its working perfectly! This is the system that gave me so much grief in the past. So now I have another system that I am bring into the network. Combine that with my fuji arriving and I am getting towards my all time high of computers that I have at home (that are functioning, I have always had various computers lying around in different states that COULD work but ….).
Finally I have also been working on a monitoring solution. I have assessed just about everything and it is down to two really, orca and cacti. Basically cacti polls via snmp and stores the result in a mysql db and then uses rrd to produce nice graphs. Orca on the other hand has a client that runs on a given machine and collects data. You then automate a cp of that data back to a central place and run orca on it that takes the data and again using rrd produces nice graphs. The key issues as I see it are as follows:
* orca
- very very flexible - can extend it to do whatever you want
- due to the nature of rsync automation, even if the network goes down or you lose your central box the data is still kept locally and pushed when it is back up. This means you never lose data regardless of what happens.
- has very detailed graphs out of the box
- its a pain to configure, a real perl hack from hell
- its a pain to extend as have to muck around with perl and horrible rrd stuff
- data is not stored in a db
* cacti
- very flexible
- php driven interface makes it easy to use / manage
- snmp polling means that if the cacti box goes down or the network fails you lose data for the hosts in question until the problem is resolved. ie nothing is cached locally as it is a snmp poll that gets the info
- data stored in db = scales / fasters / more powerful
- fairly sure it can but I don’t know HOW to make cacti do some of the really flexible stuff that orca can do

I welcome feedback on this as I am really torn which way to go.

Microkernels

April 24th, 2005

I recently had a debate with one of the other geeks I work with. She was saying that she believes MacOSX is the way forwards and that its better then linux in pretty much all area’s and that microkernels rule. It wasn’t quite that strongly worded but she was definitely having a fairly serious go at me being a linux supporter (what made it worse from my perspective is that she is / was also a big gentoo supporter so the “defection” stung more). In answer to her points I ended up writing the below email which I think some of you might be interested to read.
———————————————————-
Subject: Microkernsl

heya,

after our discussion the other day I decided that I didn’t know enough about
them, and I was also curious as to where my “instinctive” dislike of them had
come from. I was also interested in assessing your three main reasons for
loving OSx, those you stated as:

1. stability
2. it “just works”
3. security

So I set of in search of the microkernel….

The first thing I found, which partly explained my aversion was the “original”
flamewar between torvalds and ast as shown at this link:
http://people.fluidsignal.com/~luferbu/misc/Linus_vs_Tanenbaum.html
Funnily enough Linus in this old exchange suggests that he acknowledges
microkernels as potentially being better, BUT, that he hasn’t seen a decent
implementation. This wasn’t making me feel any better about attacking you on
the issue. :)

I continued looking though and found that Linus wrote a book, in which he
“apparently” attacked OSX, this seemed on topic so I tracked this down, it
turns out that Linus was misquoted, but the following link does show that he
doesn’t like microkernels:
http://www.itworld.com/News/2001/4/LWD010410maccomments/

This was the first indication that he no longer even thought microkernels are
good.

I then started to track down OSX, and was surprised to find numerous
references to the fact that OSX is NOT a microkernel. It is in fact a hybrid
kernel. Apparently one of the real issues with microkernels is their speed
compared to monolithic ones, and as a result many “microkernels” actually
start dumping a lot of userland into kernelland to speed it up, osx is no
exception and thus it can’t call itself a microkernel. This at least
clarified some of my confusion over the userland / kernelland conversation
yesterday as it was quite clear to me that OSX wasn’t a pure microkernel
implementation. A relevant link on this is:
http://www.oreillynet.com/cs/user/view/cs_msg/2719

By now I was interested in what Linus actually DID think about microkernels
and, if he had issues with them, precisely what those issues were. It was
surprisingly hard to get that information, I stumbled across things like:
http://www.ussg.iu.edu/hypermail/linux/kernel/0202.3/0464.html

which suggested that he thought they were basically too slow, wasting time
shuffling data around between rings rather then actually DOING stuff, but I
still wasn’t getting a clear picture. It was interesting to note though that
Alan Cox and some others did like one aspect of microkernels (while being
fairly derrogatory about the rest) and that is security. Apparently because
of the structured rings, and having to pass things cleanly between them, this
leads to an increase in security but at the cost of performance.

Finally I stumbled on :
http://www.sindominio.net/biblioweb/telematica/open-sources-html/node86.html

which clearly states what linux doesn’t like about microkernels and why he
took the approach he did with Linux.

Now I turned to microkernel implementations and found:
http://os.inf.tu-dresden.de/L4/LinuxOnL4/overview.shtml
which amusingly enough was a microkernel implementation on linux, and
according to every resource I could find, supposedly the best micro kernel
around.

wikipedia’s take can be found:
http://en.wikipedia.org/wiki/Kernel_(computers)
and basically says that there is still no successful microkernel
implementations.

What was interesting through all of this was that there was a lot of research
done in the early 90’s around the concept of microkernels, but despite some
papers showing (actually they are not necessarily definitive but are often
quoted as such) that microkernels could run as fast as monolithic one, no one
has EVER implemented one that could. Furthermore if you cast an eye over the
list of monolithic kernels and the list of micro kernels you probably won’t
even recognise any microkernels unless you are an old school geek, they are
just not successful. Now you might claim that they are not successful because
not many people have worked on them, but that is patently not true, a LOT of
money and time was spent trying to make them work, but they just havn’t. Now
I don’t take quite as hard line as linus on them, but it was very notable
just how few success stories I uncovered during my little search last night.
On an amusing side note about the only area where microkernels can claim
success seems to be security, and so there I have to apologise and say that I
was wrong, though I do feel it worthwhile to point out that OSX is not a
microkernel.

I think its also worthwhile pointing out that it is not just Linus that feels
this way. Almost EVERY notable, hardcore OS programmer seems to have a fairly
negative attitude towards them, interestingly enough, including people like
Theo De Raat:
http://slashdot.org/article.pl?sid=00/12/11/1455210&mode=thread

So, to go back to your statements:

1. stability.

I am sure that osx is “Stable”, or appears that way for the desktop, but then
its not really fair to compare it to linux in that context as linux is not
aimed at the desktop (at least not according to most people). Instead a much
better comparison would be to compare osx under heavy load, doing server
tasks. So i started trying to get info on where osx had been deployed in a
server environment and how it had fared. Now if you can, please point me to
information like this, because, after countless google searchs for “osx
deployment enterprise” and the like I couldn’t really find any decent
information, the best was:
http://www.macnewsworld.com/story/33706.html

So I will simply say on this point that for the desktop space it might well be
stable, but then so is Linux, though I will grant you that it probably takes
a bit more knowledge to get Linux desktop to the same level of stability /
functionality as OSX comes OOB.

2. “it just works”.
One of the interesting things you said yesterday was that there is only “one”
version of the kernel at any point in time. Given that, I pose the thought,
is it any wonder it “just works” ? I mean linux has so many ports, so many
kernel versions, its the largest project of its kind in the world and growing
at an astounding rate. I suggest that if there was ONE kernel that all
vendors used then that kernel would also “just work”, it is a direct result
of how linux is written that the kernel is comparatively fragmented and
consequently not as well tested / supported in “one” form. Also bear in mind
that osx only has to support a very limited number of hardware devices, ALL
of which apple has control over. This is in stark difference to the x86 world
of PC’s where there are incredible variety of hardware, which no one person
has control over. So yeah, OSX just works for its hardware, but then if you
have any desire / need for some other hardware, it doesn’t work at all. Again
I will also point out that linux can also “just work”, but it takes more
effort / knowledge.

3. security

There were more security implications then I thought about the microkernel
design and a lot of people seem to think this makes them more secure, these
people know way more the me so i will have to go along with what they say,
though I did find it interesting that the openbsd crowd, including theo, all
think that monolithic is just as good. Still to quickly take it into
specifics, I did some searching and osx has won a few awards in various study
groups for being the most secure OS, but something that these groups /
surveys have failed to take into account (as pointed out by others not me) is
the ratio. Ie almost all of the surveys get a list of vulns / or security
breaches at firms and tally them up and present them. What they seem to
ignore is that right now there are almost no server implementations of osx,
so naturally the pure numbers will be lower. Additionally osx has just not
got the interest that the current unix based alternatives do and consequently
hasn’t had the kind of battering that security community puts anything that
is popular through.
I would hope, and expect, that osx security is good, after all the microkernel
design lends itself to this, but as osx is not a pure microkernel there is a
real risk that some of the userland stuff they have thrown into kernelland
will have security problems and that they will have their fair share of
issues over the coming years.

sorry about the long winded reply, but I was quite interested in micro kernels
a year or so ago, and our debate rekindled that.

From robot bashing to statistics generation

April 24th, 2005

Well I have managed to structure things so that right now I have a lot of time to be learning, which as always equates to a lot of updates to Disciplina Networks. Firstly as mentioned previously we now have a webstats page which will be of interest to you people that have a site hosted with us. One of the interesting things that this highlighted for me was the fact that we have a few robots, most notably Yahoo! that is abusing our site by spidering almost everything, including large data files, almost every 3 hours. As a result of this I got around to reading up on robots.txt, which in turn meant that all our sites finally have a robots.txt which should keep some of the worse abusers in line. If you have a particular need to change the robots.txt I have given to your site then let me know and we can work something out.
I have also just finished with our new syslog solution. Basically I have something similar to Nate’s setup which you can read about at his site. So now I am a lot more comfortable with our security, not because this makes us any more secure, but rather for the first time in a while I have a few aggregated places where I can easily view and use the logs that are generated by the network. In fact I have included some real time messaging for me and will be extending that shortly so I will be alerted under all sorts of conditions. Oh the sleepless nights ;)
I mentioned on my blog that I am close to becoming a gentoo developer, which is exciting as its something that I have wanted to do for a while now, so working on a project this size will be a great learning experience for me. Finally I have been going along steadily at learning Python which I am not only enjoying but finding quite easy! Although I don’t expect that to last :)