The ToDo list for Paul Davies
26/04/07 - 27/04/07
- Taking Thursday and Friday off - to work on my own stuff.
- Start updating PTI on Monday.
11/04/07 - 14/04/07
- Working from home over Easter
04/04/07 - 05/04/07
- Put Xen Howto up. Not quite finished, but if rigorously followed anyone should be able to get xen up and running on IA64 using this.
- Still to TODO in howto
- Set up a more sophisticated virtual machine. This was a late night quickie to boot a guest.
- As far as I am concerned, getting xen up and running on ia64 is really very straight forward (just use the xen readme). The major difficulty is with elilo and the documentation for its xen support. There isn't any. I actually resorted to looking at source. That is totally unacceptable for what should be a simple install. I need to look further into elilo's xen support.
- Other stuff.
03/04/07
- Installing xen from scratch and writing HOWTO. Currently on home wiki (largely complete). Will transfer to gelato wiki at some stage towards end of week.
02/04/07
- Got sound working on laptop. ALSA is horrible.
- Did kernel upgrade on laptop and dealt with associated hassles.
26/03/07 - 30/03/07
- Reading about virtualisation (xen)
- Installed vmware workstation - created some virtual machines.
- Installed windows XP and FreeBSD on virtual machines
- Been mucking around with FreeBSD
- Tried to get sound working on my laptop (reading about ALSA)
21/03/07
- Got xen working on home itanium. My problem here was my firmware required upgrading.
19-03-07
- Continuing to accumulate info regards a howto for xen on IA64. At present I have it up and running on tartufi, but my home installation is still problematic.
08/03/07
- Paul is sick. Don't expect to be in until Monday.
- Will check in my slide to paulaner though
06/03/07
- Feeling unwell. Working from home today.
05/03/07
- Fixed up page table slide for ICE contribution.
02/03/07
- Meeting in the morning
- Not working this afternoon, attending a cousins wedding.
27/02/07 - 01/03/07
- Running a guest domain now in xen on my laptop.
- I will set xen up on IA64 and document how to do it on the wiki.
26/02/07
- Reading disk partitioning howto
- Been reading (and subsequently experimenting from) grub and lilo howtos: How and where to put bootloaders manually (via dd etc).
- Still playing with boot disk construction
- Expanding my knowledge of utilities as I go.
- A little bit of reading today about ext2.
21/02/07
- Reading the Litke patch debate with interest. I might modify the PTI depending on where the debate goes.
20/02/07
- Apply Adam Litke's patch set and comment on it.
- Having had a good look at it, I will resend the arch independent patches and CC them to this guy.
- His patches provide a partial page table interface that doesn't really clean things up. As they stand, they benefit only hugetlbfs. My patch series won't solve his problem (switching in and out different function implementations at runtime). He puts function pointers in a struct to enable him to switch in and out different implementations at runtime. I would of course love to do this kind of thing but I found the performance suffered. I can't see this patch set going anywhere.
- Will try to think of something positive to say tomorrow.
16/02/07 - 18/02/07
- Set up my workstation here to cross compile IA64 kernels so I can set it up at home. Then get netbooting going.
- Finalise my golden image.
- Figure out why I can only get text from the serial console of my home itanium and the keyboard doesn't work.
- Finish my custom built boot disk and repair my home workstation.
- I am custom building it for educational reasons.
15/02/07
- Continuing sysadmin stuff that assists me with installing xen and dealing with my home itaniums installation.
14/02/07
- Working from home tomorrow as I have a guy from telstra coming between 12 and 5PM.
13/02/07
- Spending the morning sorting out home internet problems
- Spent day on sysadmin skills
- More playing with efi. Set up serial console on my home itanium
- playing with lvm.
12/02/07
- Spent day working on Itanium sysadmin skills
- Read about system imaging. Created a golden client for my home itanium and an imageserver.
- Read about efi. Still a bit to achieve here.
- Read about and used screen for the first time. Will make use of this facility in the future.
- Recently set up a serial console from my home x86 to another machine using a NULL modem cable and minicom. I will pick up the serial console of my Itanium instead.
09/02/07
- Out to HP at 1PM to pick up the new machines.
- May be a couple of trips
- No Itanium 1s
- Madisons or better.
- May be a couple of trips
07/02/07 - 08/02/07
- Reading how to get Xen installed.
- Managed to get Xen and Linux dom0 running so far on Laptop.
06/02/07
- Got access to a tarpeian from hal. Will benchmark PTI on this box and add it to my results.
- Built a vanilla kernel on the POWER5.
- I need to fix up PTI patch series so that each patch doesn't break the compile Once I have done this I will play with trying to boot the PTI on POWER.
26/01/07 - 06/02/07
- On Holidays in Melbourne, return to work 6th February 2007
25/01/07
- Posted patch as suggested by Christolph Lammeter yesterday.
- Wrote reply for Nick Piggin.
- Reading about virtualisation during the week.
22/01/07
- Updated calendar. I am away next week. I followed the instructions to fill in the calendar correctly. It doesn't register on upcoming absences.
14/01/07
- Got some useful feedback from Peter Zijlstra, regarding the way the Linux community would expect the iterators to be dealt with. I am not surprised by his suggestions at all. It is a little reminiscent of what I had in mind at the very beginning. Hopefully this guy will show up at LCA and I can talk to him.
13/01/07
- Have sent PTI patches. They bounced overnight. I sent them again. I assume the volume of patches is causing some sort of hold up. I will get them on eventually.
12/01/07
- Finish my PTI/GPT page for LCA.
- Run SPEC for the GPT. The results won't be great.
- Feed patches tonight.
- Write my patch 00 intro
- Done my final run through of patches prior to feeding them.
11/01/07
- Going to try to finish getting patches ready for feeding on Friday at home today.
- Finalising patches all day.
- PTI arch independent files diffstated/commented and ready for feeding tomorrow.
- PTI IA64 interface diffstated/commented.
- I am not commenting the GPT, except with respect to how it fits under the interface.
- Just realised I don't have a script to do average and s.d for 386.
10/01/07
- Have a look at 2.6.20-rc4
- All patches apply
- Building kernels with a variety of different configs at the moment to make sure I haven't inadvertently busted anything.
- A few warnings to kill in the PTI.
- Starting to comment patches for feeding.
- Still going on this.
09/01/07
- I will be working at home today as I have been benchmarking one of my home machines. and I want to finish taking my final benchmarks. Also have to take dog to vet.
- Finish final patch tweaks today.
- Still picking up bugs.
- Currently making sure the pti boots on i386 with many different config options.
- now OK with 3 lvl pt covering 64G.
08/01/07
- Running SPEC on tartufi
- Finished running SPEC benchmarks for PTI - will graph them tomorrow and add to wiki.
- Possibly do some tinkering with configs to make sure inappropriate choices aren't made.
- Working on wiki page
07/01/07
- fix x86 for 3 level pt when covering 64G
- Run full SPEC benchmarks overnight for the next few nights.
06/01//07
- Fixed PTI arch independent bugs. Pass LTP on IA64 and i386
- GPT now passes LTP on IA64
- todo: boot on tutti - do next week prior to feeding.
05/01/07
- fork and mmap appear unaffected by PTI on IA64 (the problem numbers from before)
- Will run SPEC (gzip and gcc tests) today but I expect no change cause by PTI
- Running benchmarks on i386 today
- hitting on a bug in PTI that only seems to hit on i386
- Just an old bug reintoduced from cutting and pasting from old code.
- hit on another bug in PTI that turns up when running LTP on a i386
- I know what it is. Need to tweak interface.
04/01/07
- Upgrading for 2.6.20-rc3
- Done - PTI passes LTP tests
- Will be running my LCA benchmarks on this kernel.
03/01/07
- Working on GPT.
- Cleaned up the GPT - now in 12 patches. It passes lmbench but it crashes a long way into LTP. I am not going to chase this around at the moment.
- I am playing around with bargraph.pl tonight.
- Over the next few days I will start doing out the PTI for LCA page with results and the reasoning behind it.
02/01/07
- Start Benchmarking of PTI for 2.6.20-rc1.
- Start working on GPT
- Code/formating prettying up
- Add iterators necessary to pass LTP.
Prior to LCA I need to (preferably before 2 January)
- Implement a number of iterators
- Pretty up GPT formatting
- Make sure GPT survives LTP
- Make sure I can run SPEC and lmbench on top of GPT
January 2 - 15
- Feed PTI/GPT to linux-mm on the 7th
- Benchmark PTI on i386/IA64
- would like to benchmark on ppc and x86-64 as well
- Benchmark GPT
- Do out a blow by blow description of the 50 odd patches and update the LCA for PTI page
- Prepare for 5 minute lightening talk at LCA (hopefully)
Time permitting
- LVHPT and bench GPT with LVHPT as well.
23/12/06
- Moving PTI from 2.6.19 to 2.6.10
- GPT now boots on the latest kernel
- PTI arch dependent is now split
- PTI arch indepenedent is now split
- GPT is split up - but I am yet to implement 5-6 iterators (not required on boot)
- LVHPT is not included at this stage. I will leave it until the end - due to time constraints.
18/12/06 - 24/12/06
- Work on GPT
GPT now reaches boot prompt
for 2.6.19-rc6.
- Update PTI for new version.
- No word from HP regarding vanilla yet. I will leave it until tomorrow or Thursday. I want to give them a decent chance to fix it before I hassle them.
- Spend the next three days on the PTI series and GPT series for finalising them for the feed on
- I also have to implement a number of lesser used iterators for the GPT so they pass LTP.
- Work on final version of PTI for 2.6.20-rc1
- Add i386 and x86-64 to patch PTI arch independent patch set
15/12/06
- PTI is done. Working on getting GPT running under it and then breaking it up.
- I am running vanilla out to HP now, as it is out of my way (I don't live in Ryde).
- Start using ccache.
12/12/06
- Working on PTI/GPT patches from home today and Wednesday.
- Note: 11AM 420 Concord Road HP building Rhodes on Thursday
08/11/06
- Write an update of my progress and post to linux-mm. Probably post it tonight.
- Attend Peter's Linux course.
- Finish arch independent PTI today.
- Start putting GPT under the PTI
05/11/06
- There appears to be a problem with the kernel being pulled to /home/gelato/linux-2.6-hg.
- Presumably related to the autobuild breaking.
04/11/06
- Mucking around with vi, as I have always used vim.
- I am going to use vi instead of vim for a while. vim makes for a lazy vi user.
- Experiment with xargs, never used this before.
29/11/06
- Working on PTI - critical iterators done. One of these iterator abstractions is a major cause of my regression.
- Benchmarking an iterator abstraction - looking for ways to remove the regression.
- Slight rework of code organisation for the iterators will remove an inlining problem.
- Backtrack and fix up a last couple of patches.
- Slight rework of code organisation for the iterators will remove an inlining problem.
- Studying effects of various iterator abstraction approaches with caliper.
- Will seek to explain why I can't achieve zero regression over the coming days, but I have driven the cost of fork right down to less than 1% regression ATM. I am not expecting any more regression here but the patch series is still unfinished so I may yet be surprised.
28/11/06
- Registering car this morning.
27/11/06
- Worked on PTI
13/11/06
- Working on PTI
- Doing some benchmarking today
12/11/06
- Working on PTI release
- Registered for LCA
- Now Patching the first eight PTI patches in the autobuild. I have left it disconnected from CVS until later in the week.
- I will clean page_table_interface gelato CVS repo shortly (been using my own repo instead) and start checking in here to make use of the autobuild.
11/11/06
- Make current patches push to rc5.
- Put them in CVS for autobuild
- I have left the hugeTLB stuff in tact this time.
10/11/06
- Added config for PTI to kernel autobuild. (NON SIM for now)
- The kernel has moved to rc5 so I will update the patches before putting them into CVS (in the next few days) for the quilt push.
- Added myself to the absences list on the keg wiki. I did an experimental single entry first. Then did another one spanning the full week. There is a bug stopping me from removing my first test entry.
- Sent email to gernot and there are HP funds for me to attend LCA. I will enroll on Monday.
- Worked PTI patches almost to the point where I will be running my lmbench, SPEC and caliper to have a look at PTI regression.
08/11/06
TODO - over the next week or so
- 6 month plan - indicating where I will be at
- Add PTI to autobuild
- Hopefully get GPT handed over from Adam
- Work on PTI
07/11/06
- Work on home network today while paulaner is up the spout.
06/11/06
- Working on PTI for LCA
- Get documentation happening for latest release
03/11/06
- Working on PTI for LCA.
02/11/06
- Working on PTI for LCA.
01/11/06
- Working on PTI for LCA.
31/10/06
- Working on PTI for LCA. The first set of patches will be purely shuffling code around. Then I will stop and benchmark and look for deterioration.
30/10/06
- Still looking at benchmarking issues prior to launching into new PTI.
- Never got started with the PTI last week with the HP visit.
26/10/06
- For the couple of days I want to get a good start on the next PTI release as I want to be ready well before LCA. I plan to benchmark this one (and document) as I go so I can get an idea of how it is changing. This will include SPEC and lmbench. Whenever something interesting happens I will investigate with caliper.
- HP visit
25/10/06
- Pratise for PTI talk at home today.
I am expecting to have the ICE slides in front of me tomorrow
24/10/06
- Practise/prepare for Thursday.
- Work on new PTI release
23/10/06
- Set up my home moinmoin wiki
- Spent time fiddling with my CVS server setup
19/10/06
- Finally managed to get SPECcpu 1.3 running on tartufi
17/10/06
- I have a number of patches of "problem areas" for examination prepared
- page fault handler abstraction
- mlpt shuffle patch
- mlpt iterator patch
- Worked on scripting during the day - so that I can do selective tests from lmbench and do calcs on 1000's of runs.
- Build SPECcpu tomorrow
16/10/06
- Create two patches for performance examination purposes.
- Patch 1: Original page fault handler abstraction
- Patch 2: Improved page fault handler abstraction
- Examine both with caliper and see what it tells me.
- Set up private cvs repository to keep benchmarking patches
12/10/06
- Spending the next four days on my home network
- Will return to PTI release and benchmarking on Monday. Too involved to do anything else ATM
11/10/06
- Run "official" measurements with caliper on the PTI today.
- Caliper has highlighted some bugs in PTI
- FIRST - Reproduce ICE benchmarks - document on wikipage.
- ON HOLD - after discovering a PTI bug
- Abstract a page fault handler only patch.
- Benchmark with lmbench
- Run caliper measurements on it.
10/10/06
- Spending a few days at home messing with my network facilities.
- Spent couple of hours playing with caliper.
- Ran some tests for a look with caliper on the PTI.
06/10/06
- Skills development: Investigating m4, autoconf
05/10/06
Start plan (DO PRELIM TONIGHT to discuss with Adam tomorrow)
- Spend more time playing with caliper.
- Write a programs to investigate cache pollution. Look at them with caliper.
Spent time reading about caches DONE
04/10/06
Spend time getting to know caliper to assist PTI evalutation DONE - I want to spend more time just looking at the tool later in the week
- Write a couple of simple programs to demonstrate to myself some of the things that caliper can do.
- First - a simple program that has known hotspots that I can tweak. See what caliper reports.
- Second - a program where I can easily tweak expected TLB misses and use caliper to show me.
- Write a couple of simple programs to demonstrate to myself some of the things that caliper can do.
- Spent some time tonight on my LFS installation
03/10/06
- Today - fiddle about with static and dynamic libraries
DONE - to be continued though.
- Continue with my Linux from scratch installation.
DONE Will be ongoing for some time...
- Work on my 3 month plan to be posted on wiki by Friday
Deferred to tomorrow
02/10/06
- Mucking around with various programming tools
29/09/06
- Going to run sister and new baby to airport shortly.
- Start a wiki benchmarking page for the Page Table Interface
- Peter may organise a large machine over the summer
- SPEC benchmarks gcc and gzip
- Postmark
- Use HP-caliper
- Regarding the results below for Page Fault. If the spurious data points are removed and rerun
- sd becomes 0 and the Page Fault will be 2.000 us for No VHPT
- sd becomes 0 and the Page Fault will be 1.000 us for No VHPT
Benchmarking results for ICE talk
***NO PTI***
2.6.17.2 vanilla VHPT OFF, 4 level page table, 16K page
null null open signal signal fork execve /bin/sh
kernel call I/O stat fstat close install handle process process process
------------ ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.56164 2.794 0.711 5.613 0.906 4.058 154.1 787.7 4049.4
s.d. (10 runs) 0.000 0.00058 0.005 0.002 0.023 0.001 0.015 0.0 13.9 30.5
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 55.85 24.28 84.09 42.19 87.35 42.05 123.83 45.02 5806.7 1.715 2.00
s.d. 0.06 0.10 0.52 0.20 0.88 0.22 4.29 0.94 103.4 0.032 1.15
***PTI***
null null open signal signal fork execve /bin/sh
kernel call I/O stat fstat close install handle process process process
------------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.57173 2.794 0.705 5.573 0.907 4.087 157.1 778.5 4090.8
s.d. (10 runs) 0.000 0.00018 0.008 0.000 0.021 0.001 0.010 0.0 20.2 17.1
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 56.07 24.24 84.30 42.16 87.02 42.13 122.16 45.39 5978.1 1.674 1.90
s.d. 0.08 0.11 0.78 0.20 1.08 0.23 4.71 0.14 114.0 0.023 0.32
SUMMARY
2% Slow down on fork
exec and page fault slightly quicker
***NO PTI***
2.6.17.2 vanilla VHPT ON, 4 level page table, 16K page
kernel call I/O stat fstat close install handle process process process
------------ ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.56145 2.769 0.712 5.695 0.907 4.090 145.9 769.7 3990.7
s.d. (10 runs) 0.000 0.00022 0.007 0.001 0.017 0.000 0.017 0.0 24.9 16.7
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 56.08 24.41 83.91 42.21 86.54 42.24 119.25 45.50 4917.1 1.618 1.50
s.d. 0.06 0.09 0.75 0.18 1.09 0.22 3.05 0.11 41.4 0.041 1.58
***PTI***
null null open signal signal fork execve /bin/sh
kernel call I/O stat fstat close install handle process process process
---- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.57058 2.819 0.705 5.632 0.906 4.111 151.5 780.6 3984.3
s.d. (10 runs) 0.000 6.74948 0.009 0.000 0.029 0.001 0.007 0.0 14.3 16.9
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 56.18 24.53 84.84 42.35 87.29 42.36 119.74 45.55 5024.8 1.576 1.30
s.d. 0.06 0.04 0.27 0.15 0.22 0.19 2.58 0.28 39.9 0.027 0.95
SUMMARY
3.8% fork slow down
exec slightly slower
page fault slightly quicker
28/09/06
- Run Benchmarks
- I have done a run with 5 repeats. I am going to run them again with 10
- Will send "official" results to Peter for tomorrow morning
- Prelim figures: A few percent slower for fork and exec.
27/09/06
- Taking the more abstract content of the poster and putting it in the place of some of the detail in the PTI slides.
- I can't build the latest cvs checkout of the talk. I will look into this.
- Run some benchmarks on latest PTI tomorrow (I only have numbers for 2.6.12)
- Ask Adam how to get the stepwise stuff to work.
- Checked in stuff. Fine tune tomorrow.
26/09/06
- Still tinkering with diagram
- I have been off colour for the last few days. Will hopefully finalise stuff tomorrow.
25/09/06
- Make PTI slides more abstract
- Add notes to PTI slides
- Fix up the diagram
- The required diagram is not in cvs (although all the others seem to be). It is in the final diagram, but it does not exist as an eps or fig.
- Found it.
- Working from home today. Be in all day tomorrow and wednesday.
21/09/06
- Looking at Gelato ICE Talk
- Putting Adams notes in slides. First two slides done (banked in CVS)
- Didn't have time to do the two hybrids.
20/09/06
- Play with Latex
- Done 4 PTI slides. Will do one diagram - not done yet.
- Will check into CVS tonight.
19/09/06
- Picking up Itanium 1 at 5:30 today.
- OMG it is huge. ~70kg. I am going need some assitance to get this thing out of my car and onto a trolly.
- Work on talk - plan diagrams.
- Do the MLPT vrs GPT slides. plan diagrams etc. (At the moment I am doing PTI only). I am bogged down finding out about a bunch of utilities - pspresent, chaksem, texpower, seminar ...
- I will check in the PTI slides ready for Thursday. Will come in and discuss the talk with Adam and Ian on Thursday.
- Look at Ians stuff tonight.
- Checked out and built the talk.
18/09/06
- Work on the PTI/GPT/Superpage talk for Gelato ICE
- Started. There seem to be problems with weill at the moment so I have been unable to check out the talk.
- Installed some of the tools pspresent, texpower. will muck around with them tomorrow.
- Made contact with Chris from Aurema. Will pick it up some time this week.
15/09/06
- Paul still sick. Back Monday
14/09/06
- Paul is ill at the moment
- Will organise to pick up machine from Aurema
11/09/06
Looking at the detail required in for the minute taking. DONE
Fixing up to email in. DONE
- Do late progress report.
I am fiddling with torrone ATM. DEFERRED
- Further aquainting myself with our network.
Have a last go at the abstract. DEFERRED
Work on the abstraction part of the talk. DEFERRED
- Spent the afternoon thinking about VLA's, self mapped page tables and Ian's diagram of a VLA.
- Discussed VLAs in depth with Ian. Discussed what is a cache and what is a TLB.
I confirmed my understanding of the use of the TLB in this regard with Matt Chapman.
- Will do an enhanced diagram to illustrate a VLA. - there is a bit missing in my opinion.
08/09/06
- Ian helped me straigten out my misunderstanding with respect of the a self mapping page table and the virtual linear array.
06/09/06
- Mucking around with CentOS Live cd.
- There is a CentOS 4.4 for i386 but there only appears to be a version 4.3 for ia64.
- I am burning the 4 cds for ia64 version 4.3.
01/09/06
- Read about the demos for the CSE open days.
- Meeting at 4PM regarding this.
- Reading again about the short format VHPT.
- OMG I finally understand it. That didn't take long did it.
31/08/06
- I tried to build the latest LVHPT patches into ~ 2.6.15 I could compile and not boot it (I couldn't find the right config to build it with).
- Rather than ask Ian for a config I decided to retackle ivt.S, the long and short format VPHT etc etc. I started to read Matt Chapmans LVHPT paper (I have advanced a lot since I last tried to read it). I am in the process of doing this. Spent my night revisiting the short format of the page table walker in the IA64 text.
30/08/06
- Work on applying LVHPT So LVHPT (in gelato CVS) is only up to date for 2.6.14. Not surprisingly these patches don't apply to 2.6.17.2.
- I am sure Ian has more recent versions that this tucked away because this stuff is 11 months old and FIND_PTE etc are in defines still not macros.
- Spoke to Ian. He has pointed me to the lastest. It is going to take some effort to get the patches to work with 2.6.17.2.
- Obviously I know how to patch stuff etc. But there are a bunch of manouvres that you must be able to do but don't know how. I am going to put some time into more advanced patching skills to make my life a bit easier.
29/08/06
- Working from home today (I have a dentists appointment in the middle of day)
- I have a list of things to work on given to me by Adam
- Will start working on getting the thing to work will LVHPT (NOT DONE worked on broadening Linux skills instead).
28/08/06
- Get PTI patches to Adam so I can play with kernel modules (send patches)
22/08/06
- Get updated PTI to Adam
- Pull Ians git tree and have a look at his page table walking stuff.
- Got Ian page population module. Got it working. Plan to add some extensions to it next week.
- Adam wants an analysis of pud and pmd levels that contain single pointers through to the next level (where guards will benefit).
21/08/06
- The kernel GPT is being looked up OK. Investigate the liklihood that lookup of user GPT's is failing.
- This is definitely the problem.
18/08/06
- Get synced GPT to Adam by end of day in some form. unless I hit another problem.
- Tonight look at debugfs
17/08/06
- Spend the day on my backtracked GPT patches.
- Not necessary. I found my problem and am getting back on track
16/08/06
- Working on gpt patches.
Still being mystified by a break down. Backtracking here due to frustration
- Mucking around with modules (will talk to Ian but I want to play with modules)
Read about relayfs Now know how to turn it on and what it is for
- Always built monolithic kernels for ia64 in the past. I have created a trivial module and am still in the process of getting it to load.
12/08/06
- Send abstract to Peter
- Working from home today
- Work on GPT in kernel.
- Also look at the framework required to write a kernel module.
- Adam suggested writing a module to gather data page tables use of the address space.
Talk to Ian, discussing this with him on Monday he also said he'd take a look at it - AdamWiggins.
11/08/06
- Write abstract for ICE by the end of the day.
- - approx 10 lines.
- Tidy up the userspace GPT and give to Adam.
10/08/06
- Got Adams original GPT to work with new config.
No anomoloy - just a config problem fortunately
- My synced GPT is breaking down - figure out why
- GPT is reading lookup now. Get stuff tidied up.
09/08/06
- GPT compiled into kernel under full PTI.
- I just need to get it to boot now.
- Look at kerncomp again.
- Back doing further reading on my bash scripting skills.
- Playing around with my understanding of permissions
- It appears to have broken with the change of kernel from rc3 to rc4.
- Is this a coincidence?
- I a switch user to ianw I cannot use sudo - I get pam - authentication failure each time
- On the final set of results the summary.log records the kernel version as rc3 and the git-pull says it pulled the patch for rc4.
08/08/06
- I have been putting Adams GPT under latest PTI and parts of the PTI that need to change have become intermingled with the GPT. It has become a bit of a mess.
- I am going to go back and adjust the IA64 PTI to fit in with Adams stuff
- I have most of the gpt headers compiling in the kernel as intended
- Should have it running by Friday.
07/08/06
- Doctors appointment in gladesville at 11AM then I will be in.
- Working on userspace model etc.
- You will just be able to run a script to update the GPT in the kernel from userspace. ie. develop in userspace, run script and compile for kernel.
- Making a couple of adjustments to PTI to better slip in adams stuff.
- Update wiki for PTI and planning.
04/07/06
- Will do some work tonight. I spent my afternoon chasing around some laptop problems (had to return to the shop).
02/07/06
- Ask Ian to have a look at his cron table to see why the kern comp script
isn't being run. DONE
- I have create, build and lookup compiling without fixing mem allocator. Fix mem alloc and see how it goes. - OK. I see it is not using the quicklists yet.
Starting to read through some of the core GPT code to refresh my memory. STARTED
I can build and lookup the GPT in userspace. DONE
Write an email to Chris Wright about the iterators. DONE
01/07/06
- Continue working on userspace version of new GPT.
- Build a little quicklist allocator for the model.
- Core GPT building in user space - #defining out the allocation still.
- Will start to bring in the interface functions.
- Have another look at the kernel build. Apparently Ian has a cron job for the kerncomp script and it is OK.
- It is an internal problem to do with Ians crontab and permissions.
31/06/06
- Start with looking at why the new GPT isn't booting on tartufi.
- Tested to see if it boots UP - NO.
- It appears to break on the first lookup to me - not related to the dormant race condition.
- Building userspace version.
- Ditch a bunch of unused kernels from home directory before I get shamed.
28/06/06
Go over the IDA64 PTI patches with Adam DONE
- Small changes will be made to PTI to make it fit in with Adams stuff as best possible.
- See if I can't figure out why the assembler lookup is barfing (Adam speak) on tartufi, or most importantly get it to boot by reinserting a variation of the old lookup.
- Probably come up with a new short term plan and some goals.
- Start syncing the GPT under the latest patches.
27/06/06
- Get new GPT patch from Adam and put it in the latest PTI series.
- This will take some time and effort
- Look to see why kernel autobuild isn't running.
- Had a quick look at it. The kerncomp.sh script isn't being run. I don't know why - I would start by looking at the crontab but I haven't got the necessary permissions to investigate any further.
- I don't see any cron job being run by the system-wide crontab. I assume it is being run from ian's crontab.
25/06/06
- Work on final disentanglement
- Now just need to do some cleaning up.
- Working on bash and environment skills.
24/06/06
- Work on IA64 patches
- Working on disentangling the various pgtable-nopud.h pgtable.h abstractions.
22-23/06/06
- Personal development this weekend.
- Reading through the "Cathedral and the Bazaar" - relevant to me because I tend to dither and not want to show anything to people until it is perfect. This greatly reduces my effectiveness. I am still learning to "release early and release often".
- Learning about Linux from Scratch. Plan to follow this through.
- Building networks at home to improve networking skills.
20/06/06
- The collective PTI patch series should be far more stable for a while.
- Do out new prepare GPT patch series - to sync with Adam.
- A stable PTI will enable me to attack performance (which is actually interesting).
19/06/06
Finish IA64 patches and send to Adam today. FINISHED PATCHES
- Do some extra testing prior to sending to Adam
Get PTI to work for x86-64 (so Lee can test it on one of his Opterons). DONE
- Reading.
- Reading about caching.
- Reading about different architectures - about gaining a better understanding of where different architectures fit into the scheme of things.
18/06/06
- I have an appointment with my ENT doctor at 2PM in chatswood.
- Working on IA64 patches for Adam.
17/06/06
Work towards resyncing with Adam High Priority.
- Preparing IA64 patches ASAP so Adam can move again.
- Helped Adam get rid of a legacy assembler hack created by myself.
- Look at PTI performance issues for a small amount of time each day.
- Working on sysadmin skills tonight.
14/07/06
- Think about including HugeTLB in PTI. I left it out initially because it made life easier.
- Work on PTI IA64 patch series.
- Review some theory and get to work on Lee's feedback email.
- Had a look at the external time command.
13/07/06
- Reposted patches to linux-mm. No matter how hard I tried, I could not seem to use the smtp.cse.unsw.edu.au server to send my mails.
10/07/06 - 11/07/06
- Having operation. Indisposed.
09/07/06
Booted final patches on tartufi DONE
Booted final patches on tutti DONE
Running LTP on tartufi. DONE
Sent patches to linux-mm Haven't come thru yet.
08/07/06
Finished splitting up the patches DONE.
- Need to write out the explanation for each patch - then feed.
- Explanations and mails ready.
- Do final test tomorrow and then hit send.
07/07/06
- I managed to get 2.6.17.2 vanilla running on tutti.
- self introduced bug stuffed me up.
- Want to start the split today.
- I will make sure the patch split PTI runs on tutti
- Bug in vmalloc_to_page - here is where to start. This is breaking the PTI on tutti.
The bug is in lookup_page_table (an absolute howler)
PTI now boots on tutti. Will send bug fix to Lee.
- Doing out patch split now.
06/07/06
- Building PTI on tutti.
- PTI crashing tutti - incorrect conclusion. I thought I was booting my kernel when I wasn't. In actualy fact - I still haven't got a 2.6.17.2 kernel booting yet. I have at least got it booting my kernel but it breaking down. Something wrong with the config.
05/07/06
Playing with sendpatchset utility Piece of cake.
- Lee is going to run the patches on an Olympia some time this week.
- Should be interesting. The patch is untested on a NUMA machine. I have only ever run UP or SMP. Hope it doesn't break.
04/07/06
- Working on IA64 dependent interface today.
Send PTI to Lee S. DONE
03/07/06
- Working from home today. I want to finish PTI today and get it to Adam for tomorrow and then straight to Lee.
- Actually I am just going to send PTI to Adam for assessment now. It is 90% done. Then I will send it straight onto Lee when finished.
- More or less finished it. Will send to Adam shortly. Expect to send it to Lee some time tomorrow.
02/07/06
- Getting PTI ready for Adam/Lee.
01/07/06
- Working on PTI for release
- I want to get them to Lee within the next few days.
- Starting to look good. Picked up a number of bugs in fremap.c and filemap_xip.
- I expect to have the PTI ready for Adam on Monday. When Adam has reviewed it, I will send it straight through to Lee. I will then split the patches out pending advice from Adam and Lee
30/06/06
- Building up a next release for the latest kernel.
- Leaving performance for now.
28/06/06
- I have been trying to measure various aspects of the PTI that have slowed down forking. I created a bash and perl script to to run the lmbench fork test a chosen number of times and take the average and standard deviation.
- I am running the fork test with a sample space of 10000. Avg: 143.86 s.d 2.630 ~95% accuracy (2 s.d) 143.86 +- 2x2.63/100 Expected +- 143.86 +- 0.05
- I am now running the fork tests for some individual abstractions that I am interested in. I am studying the cost of an individual
abstraction (pte_alloc). Avg: 144.63 s.d 2.7
- This looks to me like simply moving a function (called critically in memory.c during fork is having a significant impact on performance. Quite alarming to me.
24-27/06/06
- Start rebuilding PTI
- Measuring causes of slow downs.
- I now have very interesting and useful results. I am going to pursue the performance issue on Thursday.
- I will benchmark on x86, as I am suspicious that it something unique to the IA64 architecture causing me problems.
- Measuring causes of slow downs.
23/06/06
- Working on my benchmarking/scripts etc to slash my time wasting.
- I have finished a little set of bash/perl scripts that will save me a huge amount of time.
- bash scripting improving.
- I will be building on these scripts in the near future.
22/06/06
- Mucking around with q-syscollect. I want to do some profiling to see where fork spends its time.
- Figured out more or less how to use it (had a play with it) - will start to make use of this tool in future.
- Work on some scripts for automating benchmarks this afternoon.
- I want my own framework for running benchmarks on tartufi overnight.
21/06/06
- Fix up a couple of known bugs in official patch set.
- Fix up some of the hangover stuff from the name changing.
- Remove function pointer calls from special iterators.
- Demonstrated the principle for copy page range. Now I want to measure the cost of passing functioning pointers before I make any solid decisions about what I do. I am going to produce some patch sets for benchmarking (which will involve x86 and IA64). I just want to get an idea of how costly passing the function pointer is w.r.t my performance problems.
- I also need to do some scripts to automate my benchmarking.
20/06/06
- Continue tidy up of PTI patch set.
Finish fixing the naming scheme for page table files. DONE
Make sure you can at least compile a kernel with the GPT running. DONE
I plan to give Adam a new set of PTI and GPT patches by the end of today. DONE
Stuffed up the patches I sent to Adam. I have been working on this and will resend soon. DONE
Remove passing of function pointers to critical iterators to get back performance. MOVED TO TOMORROW Critical iterators at the moment are earmarked as copy_page_range, unmap_page_range and the free page table funcs.
18/06/06
- I am going to work most of the night as I expect to be up until 4am and not in tomorrow.
- I have run some tests. There is no catastrophic memory leak (as OOM suggests). I need to work out why build_page_table is breaking. - getting a GPT_OVERLAP.
- I have NOW reached the boot prompt. It breaks on very low radices though. (Probably an internal locking issue). I am not going to chase it down now.
- Working on PTI for release late in the week (or thereabouts).
- Fixed mm_struct typedef problem.
- Fixing the file naming problems. Unfortunately it reverberates through the whole patch split.
16/06/06
- Still debugging.
- Build_page_table is breaking causing an erroneous OOM error. It is not really out of memory. I must have stuffed the page table translating it accross somehow.
15/06/06
- Unfortunately I am picking up a number of abstraction errors.
- I will be spending today fixing what I have identified so far. Presumably one or more of these is causing me the problems in the boot process of the GPT.
- During boot, I get "VM: killing process init". I presume this means I am running out of memory (massive memory leak) - which I assume is caused by unmap_page_range.
- Picked up ridiculous errors (introduced by me) into unmap,vmap,copy_page_ranage
and change protection iterators. Must be tired
14/06/06
- I will spend some more time trying to nail some problems on GPT patches.
- Still hunting the problem. I will devote the rest of the day to this.
- Picked up naughty abstraction error in unmap_page_range. Presumably some more to come.
13/06/06
- I have enough for Adam to start working on the GPT, in the kernel. I am going to sort out some patches for him (need to do some tidying).
11/06/06
- I have enough for Adam to start working on the GPT, in the kernel. I am going to sort out some patches for him (need to do some tidying).
10/06/06
- Get GPT going under 2.6.17-rc5.
Compile in GPT implementation DONE- now I need to call it and see how it goes.
Reading assembler lookup DONE
Up to create_user_page_table DONE
- Going OK. Still some work to do.
08/06/06
- Paul is sick at the moment
07/06/06
- Working from home today - I have a bad cold and can't stop sneezing.
06/06/06
- Continue getting GPT patch for Adam
- I have been creating the "prepare for GPT patch".
I will incorporate appropriate elements of this back into the PTI series.
The original PTI series did too much rearranging of stuff confusing the kernel writers. This patch will largely shift back into the PTI for the next PTI version. GPT Prepare patch DONE - Send prepare patch to Adam. It will make it very clear what needs to be done - where he needs to look at etc.
05/06/06
- Concentrate on getting the GPT to run under the old interface.
30/05/06
- Put patches in CVS
- Running benchmarks.
- Feed them tonight.
- Visiting my new born nephew today.
29/05/06
- Benchmarked rc5. 3.5% deterioration in fork. Much better than than the results for 2.6.12.
What was the deterioration for 2.6.12? - AdamWiggins
- No deterioration in mmap latency.
24/05/06
- Working on re-building a full PTI. Going well. Organising patch split as I build it up. Probably feed on weekend.
I have all my patches sorted out up to iterators. I have to actually go ahead and do the split (trivial).
- Enquire to postmaster about patch size limit. 100 000 characters doesn't seem right. Think they mean 10 000.
- Test on i386 tonight - forgot to do my Kconfig for i386. Should do for powerpc as well. before feed.
23/05/06
- Continuing preparation for patch feeding. Very time consuming.
- Also found a several problems with the lmbench sum perl scripts which should be fixed. eg: Failure to calculate s.d. on fork.
- Will start patch split tonight. Do the first three or four patches.
- Patch 0). The big explanation patch. Defer this guy.
- Patch 1). Introduce basic interface.
22/05/06
- I spent my weekend trying to work out how to get the loss in peformance back in the page fault handler. I am pushing on with this today. The interface will change according to the solution to this problem.
- After talking with Adam, I have decided to just go with one of my solutions and work on performance after feeding.
19/05/06
- I cannot resist continuing to try and get to the bottom of PTI performance problems. Benchmarking various ideas at the moment.
- Prepare patches for feeding - architecture dependent and independent.
- Talked to Adam about why I was still losing a little bit on fork. He gave me several things to think about - try them today. This time I am certain all performance overhead will be removed.
- Moving the locking inside the implementation in the page fault handlers to facilitate fine grained locking support into the PTI.
18/05/06
- I patched the user level driver stuff yesterday but it didn't compile Peter has had since fixed this patch set for the latest kernel so I will have a look at it and see where I went wrong.
- As most of the autobuild is now OK I will fix the page fault
handler patches and document it on the page table interface page. LOSS of performance in page fault handler is now 0.5% for fork, nothing for mmap - BIG improvement Actually - I need to work out why fork is still slowed down. I was expecting none at all.
17/05/06
Patch user driver stuff and put in CVS TODO - put in CVS
Fix up the configs for the i2 - uniprocessor and preemptible DONE
- Change firmware for tartufi and cassata
I have deferred this to the weekend. I have asked Ian not to do it. I realise it is a five minute job - but I want to read about EFI etc first and do
- Spent some time this afternoon talking with Adam about just why they do pass pmds around the page faulting functions and relook them up. (Due to highmem on i386)
16/05/06
- Experimented with the build scripts tonight. I have been migrating the scripts across to work in my own directories - just to have a fiddle with them. I don't have the permissions to work in situ. Need to fix this. I think we should shift the kernel autobuilding out of ians home directory into a general gelato directory.
- I am 100% sure I know exactly what I am doing and should be able to sort it no problems. (OK - I had a minor mishap - grrr cut and paste error)
- Making good progress - I am cloning/fetching the latest kernel correctly. I am rebuilding them back up, it is the easiest way to learn bash. I haven't got it to build successfully yet - but I want to go to bed.
- TODO: Update the wiki page on the kernel autobuild
- I was running my experimental scripts (partially built) and my output was written to the real scripts output. (I cut and paste - forgot to cut). The real scripts have now been run and new output is being generated.
- LESSON: Should not have experimented with scripts on eisbock. Too easy to make a mistake.
I might add my pti patches to the autobuild - This is definately worthwhile (AdamWiggins)! (OK PaulDavies)
- Investigation why user level drivers patch doesn't apply in autobuild.
System calls have been added into the ia64 syscall patch. Easy to fix.
Similarly other syscalls have been added to i386. Easy to fix.
- Investigate why two kernels failed to build. Work out the config conflicts.
- Have a look at why vhpt-sim patches don't apply. Get from lists.
15/05/06
- I will be a little late as I have a doctors appointment at 9:20 in Chatswood.
- I am going to look into loss of performance in the page fault functions today
- I have a 3% deterioration in fork due to page fault abstractions alone. I have been aware of this for some time and have a solution in mind. I will see how it goes.
- Start on the problem of fixing the kernel autobuilding. Finding out a little about git and cogito. The version of git has now been upgraded to the latest version.
07/05/06
CeBit tomorrow.
Add a wiki page to explain how to get a kernel up and going - PaulsKernelBuild
- Reading about vNUMA and linux on linux.
06/05/06
- Reading about basic locking primatives.
- Reading about memory barriers.
- Examined the reader/writer concurrency problem. Wrote a program using the pthread libraries to demonstrate and understand the problem in my mind. - still in progress but I expect to finish it tomorrow.
05/05/06
- Working on my in depth understanding of the page table in general
- There is a lot of discussion on linux-mm about a possible page table walking problem as a result of calling free_pg_tables. It is providing a very informative discussion on the page tables in general.
- Learn about memchecker, PIN, memtoy and caliper.
04/05/06
- Read about Nick Piggins lockless radix tree.
03/05/06
- Have doctors appointment at 9. Will be in after that.
02/05/06
- Started to look at the finer details of some of the locking in VM.
- Working on documentation
01/05/06
- Move arch independent interface to 2.6.17-rc3. DONE
28/04/06
- Sent the architecture independent interface to Adam for latest kernel so he can look at it on the weekend.
27/04/06
- Very happy with PTI (architecture independent) - will post to CVS tomorrow.
- I have two custom dual iterators (copy_page_range and move_page_tables).
- I have five iterations that call a general read iterator
- I have two customised read it