The ToDo list for Paul Davies
26/04/07 - 27/04/07
- Taking Thursday and Friday off - to work on my own stuff.
- Start updating PTI on Monday.
11/04/07 - 14/04/07
- Working from home over Easter
04/04/07 - 05/04/07
- Put Xen Howto up. Not quite finished, but if rigorously followed anyone should be able to get xen up and running on IA64 using this.
- Still to TODO in howto
- Set up a more sophisticated virtual machine. This was a late night quickie to boot a guest.
- As far as I am concerned, getting xen up and running on ia64 is really very straight forward (just use the xen readme). The major difficulty is with elilo and the documentation for its xen support. There isn't any. I actually resorted to looking at source. That is totally unacceptable for what should be a simple install. I need to look further into elilo's xen support.
- Other stuff.
03/04/07
- Installing xen from scratch and writing HOWTO. Currently on home wiki (largely complete). Will transfer to gelato wiki at some stage towards end of week.
02/04/07
- Got sound working on laptop. ALSA is horrible.
- Did kernel upgrade on laptop and dealt with associated hassles.
26/03/07 - 30/03/07
- Reading about virtualisation (xen)
- Installed vmware workstation - created some virtual machines.
- Installed windows XP and FreeBSD on virtual machines
- Been mucking around with FreeBSD
- Tried to get sound working on my laptop (reading about ALSA)
21/03/07
- Got xen working on home itanium. My problem here was my firmware required upgrading.
19-03-07
- Continuing to accumulate info regards a howto for xen on IA64. At present I have it up and running on tartufi, but my home installation is still problematic.
08/03/07
- Paul is sick. Don't expect to be in until Monday.
- Will check in my slide to paulaner though
06/03/07
- Feeling unwell. Working from home today.
05/03/07
- Fixed up page table slide for ICE contribution.
02/03/07
- Meeting in the morning
- Not working this afternoon, attending a cousins wedding.
27/02/07 - 01/03/07
- Running a guest domain now in xen on my laptop.
- I will set xen up on IA64 and document how to do it on the wiki.
26/02/07
- Reading disk partitioning howto
- Been reading (and subsequently experimenting from) grub and lilo howtos: How and where to put bootloaders manually (via dd etc).
- Still playing with boot disk construction
- Expanding my knowledge of utilities as I go.
- A little bit of reading today about ext2.
21/02/07
- Reading the Litke patch debate with interest. I might modify the PTI depending on where the debate goes.
20/02/07
- Apply Adam Litke's patch set and comment on it.
- Having had a good look at it, I will resend the arch independent patches and CC them to this guy.
- His patches provide a partial page table interface that doesn't really clean things up. As they stand, they benefit only hugetlbfs. My patch series won't solve his problem (switching in and out different function implementations at runtime). He puts function pointers in a struct to enable him to switch in and out different implementations at runtime. I would of course love to do this kind of thing but I found the performance suffered. I can't see this patch set going anywhere.
- Will try to think of something positive to say tomorrow.
16/02/07 - 18/02/07
- Set up my workstation here to cross compile IA64 kernels so I can set it up at home. Then get netbooting going.
- Finalise my golden image.
- Figure out why I can only get text from the serial console of my home itanium and the keyboard doesn't work.
- Finish my custom built boot disk and repair my home workstation.
- I am custom building it for educational reasons.
15/02/07
- Continuing sysadmin stuff that assists me with installing xen and dealing with my home itaniums installation.
14/02/07
- Working from home tomorrow as I have a guy from telstra coming between 12 and 5PM.
13/02/07
- Spending the morning sorting out home internet problems
- Spent day on sysadmin skills
- More playing with efi. Set up serial console on my home itanium
- playing with lvm.
12/02/07
- Spent day working on Itanium sysadmin skills
- Read about system imaging. Created a golden client for my home itanium and an imageserver.
- Read about efi. Still a bit to achieve here.
- Read about and used screen for the first time. Will make use of this facility in the future.
- Recently set up a serial console from my home x86 to another machine using a NULL modem cable and minicom. I will pick up the serial console of my Itanium instead.
09/02/07
- Out to HP at 1PM to pick up the new machines.
- May be a couple of trips
- No Itanium 1s
- Madisons or better.
- May be a couple of trips
07/02/07 - 08/02/07
- Reading how to get Xen installed.
- Managed to get Xen and Linux dom0 running so far on Laptop.
06/02/07
- Got access to a tarpeian from hal. Will benchmark PTI on this box and add it to my results.
- Built a vanilla kernel on the POWER5.
- I need to fix up PTI patch series so that each patch doesn't break the compile Once I have done this I will play with trying to boot the PTI on POWER.
26/01/07 - 06/02/07
- On Holidays in Melbourne, return to work 6th February 2007
25/01/07
- Posted patch as suggested by Christolph Lammeter yesterday.
- Wrote reply for Nick Piggin.
- Reading about virtualisation during the week.
22/01/07
- Updated calendar. I am away next week. I followed the instructions to fill in the calendar correctly. It doesn't register on upcoming absences.
14/01/07
- Got some useful feedback from Peter Zijlstra, regarding the way the Linux community would expect the iterators to be dealt with. I am not surprised by his suggestions at all. It is a little reminiscent of what I had in mind at the very beginning. Hopefully this guy will show up at LCA and I can talk to him.
13/01/07
- Have sent PTI patches. They bounced overnight. I sent them again. I assume the volume of patches is causing some sort of hold up. I will get them on eventually.
12/01/07
- Finish my PTI/GPT page for LCA.
- Run SPEC for the GPT. The results won't be great.
- Feed patches tonight.
- Write my patch 00 intro
- Done my final run through of patches prior to feeding them.
11/01/07
- Going to try to finish getting patches ready for feeding on Friday at home today.
- Finalising patches all day.
- PTI arch independent files diffstated/commented and ready for feeding tomorrow.
- PTI IA64 interface diffstated/commented.
- I am not commenting the GPT, except with respect to how it fits under the interface.
- Just realised I don't have a script to do average and s.d for 386.
10/01/07
- Have a look at 2.6.20-rc4
- All patches apply
- Building kernels with a variety of different configs at the moment to make sure I haven't inadvertently busted anything.
- A few warnings to kill in the PTI.
- Starting to comment patches for feeding.
- Still going on this.
09/01/07
- I will be working at home today as I have been benchmarking one of my home machines. and I want to finish taking my final benchmarks. Also have to take dog to vet.
- Finish final patch tweaks today.
- Still picking up bugs.
- Currently making sure the pti boots on i386 with many different config options.
- now OK with 3 lvl pt covering 64G.
08/01/07
- Running SPEC on tartufi
- Finished running SPEC benchmarks for PTI - will graph them tomorrow and add to wiki.
- Possibly do some tinkering with configs to make sure inappropriate choices aren't made.
- Working on wiki page
07/01/07
- fix x86 for 3 level pt when covering 64G
- Run full SPEC benchmarks overnight for the next few nights.
06/01//07
- Fixed PTI arch independent bugs. Pass LTP on IA64 and i386
- GPT now passes LTP on IA64
- todo: boot on tutti - do next week prior to feeding.
05/01/07
- fork and mmap appear unaffected by PTI on IA64 (the problem numbers from before)
- Will run SPEC (gzip and gcc tests) today but I expect no change cause by PTI
- Running benchmarks on i386 today
- hitting on a bug in PTI that only seems to hit on i386
- Just an old bug reintoduced from cutting and pasting from old code.
- hit on another bug in PTI that turns up when running LTP on a i386
- I know what it is. Need to tweak interface.
04/01/07
- Upgrading for 2.6.20-rc3
- Done - PTI passes LTP tests
- Will be running my LCA benchmarks on this kernel.
03/01/07
- Working on GPT.
- Cleaned up the GPT - now in 12 patches. It passes lmbench but it crashes a long way into LTP. I am not going to chase this around at the moment.
- I am playing around with bargraph.pl tonight.
- Over the next few days I will start doing out the PTI for LCA page with results and the reasoning behind it.
02/01/07
- Start Benchmarking of PTI for 2.6.20-rc1.
- Start working on GPT
- Code/formating prettying up
- Add iterators necessary to pass LTP.
Prior to LCA I need to (preferably before 2 January)
- Implement a number of iterators
- Pretty up GPT formatting
- Make sure GPT survives LTP
- Make sure I can run SPEC and lmbench on top of GPT
January 2 - 15
- Feed PTI/GPT to linux-mm on the 7th
- Benchmark PTI on i386/IA64
- would like to benchmark on ppc and x86-64 as well
- Benchmark GPT
- Do out a blow by blow description of the 50 odd patches and update the LCA for PTI page
- Prepare for 5 minute lightening talk at LCA (hopefully)
Time permitting
- LVHPT and bench GPT with LVHPT as well.
23/12/06
- Moving PTI from 2.6.19 to 2.6.10
- GPT now boots on the latest kernel
- PTI arch dependent is now split
- PTI arch indepenedent is now split
- GPT is split up - but I am yet to implement 5-6 iterators (not required on boot)
- LVHPT is not included at this stage. I will leave it until the end - due to time constraints.
18/12/06 - 24/12/06
- Work on GPT
GPT now reaches boot prompt
for 2.6.19-rc6.
- Update PTI for new version.
- No word from HP regarding vanilla yet. I will leave it until tomorrow or Thursday. I want to give them a decent chance to fix it before I hassle them.
- Spend the next three days on the PTI series and GPT series for finalising them for the feed on
- I also have to implement a number of lesser used iterators for the GPT so they pass LTP.
- Work on final version of PTI for 2.6.20-rc1
- Add i386 and x86-64 to patch PTI arch independent patch set
15/12/06
- PTI is done. Working on getting GPT running under it and then breaking it up.
- I am running vanilla out to HP now, as it is out of my way (I don't live in Ryde).
- Start using ccache.
12/12/06
- Working on PTI/GPT patches from home today and Wednesday.
- Note: 11AM 420 Concord Road HP building Rhodes on Thursday
08/11/06
- Write an update of my progress and post to linux-mm. Probably post it tonight.
- Attend Peter's Linux course.
- Finish arch independent PTI today.
- Start putting GPT under the PTI
05/11/06
- There appears to be a problem with the kernel being pulled to /home/gelato/linux-2.6-hg.
- Presumably related to the autobuild breaking.
04/11/06
- Mucking around with vi, as I have always used vim.
- I am going to use vi instead of vim for a while. vim makes for a lazy vi user.
- Experiment with xargs, never used this before.
29/11/06
- Working on PTI - critical iterators done. One of these iterator abstractions is a major cause of my regression.
- Benchmarking an iterator abstraction - looking for ways to remove the regression.
- Slight rework of code organisation for the iterators will remove an inlining problem.
- Backtrack and fix up a last couple of patches.
- Slight rework of code organisation for the iterators will remove an inlining problem.
- Studying effects of various iterator abstraction approaches with caliper.
- Will seek to explain why I can't achieve zero regression over the coming days, but I have driven the cost of fork right down to less than 1% regression ATM. I am not expecting any more regression here but the patch series is still unfinished so I may yet be surprised.
28/11/06
- Registering car this morning.
27/11/06
- Worked on PTI
13/11/06
- Working on PTI
- Doing some benchmarking today
12/11/06
- Working on PTI release
- Registered for LCA
- Now Patching the first eight PTI patches in the autobuild. I have left it disconnected from CVS until later in the week.
- I will clean page_table_interface gelato CVS repo shortly (been using my own repo instead) and start checking in here to make use of the autobuild.
11/11/06
- Make current patches push to rc5.
- Put them in CVS for autobuild
- I have left the hugeTLB stuff in tact this time.
10/11/06
- Added config for PTI to kernel autobuild. (NON SIM for now)
- The kernel has moved to rc5 so I will update the patches before putting them into CVS (in the next few days) for the quilt push.
- Added myself to the absences list on the keg wiki. I did an experimental single entry first. Then did another one spanning the full week. There is a bug stopping me from removing my first test entry.
- Sent email to gernot and there are HP funds for me to attend LCA. I will enroll on Monday.
- Worked PTI patches almost to the point where I will be running my lmbench, SPEC and caliper to have a look at PTI regression.
08/11/06
TODO - over the next week or so
- 6 month plan - indicating where I will be at
- Add PTI to autobuild
- Hopefully get GPT handed over from Adam
- Work on PTI
07/11/06
- Work on home network today while paulaner is up the spout.
06/11/06
- Working on PTI for LCA
- Get documentation happening for latest release
03/11/06
- Working on PTI for LCA.
02/11/06
- Working on PTI for LCA.
01/11/06
- Working on PTI for LCA.
31/10/06
- Working on PTI for LCA. The first set of patches will be purely shuffling code around. Then I will stop and benchmark and look for deterioration.
30/10/06
- Still looking at benchmarking issues prior to launching into new PTI.
- Never got started with the PTI last week with the HP visit.
26/10/06
- For the couple of days I want to get a good start on the next PTI release as I want to be ready well before LCA. I plan to benchmark this one (and document) as I go so I can get an idea of how it is changing. This will include SPEC and lmbench. Whenever something interesting happens I will investigate with caliper.
- HP visit
25/10/06
- Pratise for PTI talk at home today.
I am expecting to have the ICE slides in front of me tomorrow
24/10/06
- Practise/prepare for Thursday.
- Work on new PTI release
23/10/06
- Set up my home moinmoin wiki
- Spent time fiddling with my CVS server setup
19/10/06
- Finally managed to get SPECcpu 1.3 running on tartufi
17/10/06
- I have a number of patches of "problem areas" for examination prepared
- page fault handler abstraction
- mlpt shuffle patch
- mlpt iterator patch
- Worked on scripting during the day - so that I can do selective tests from lmbench and do calcs on 1000's of runs.
- Build SPECcpu tomorrow
16/10/06
- Create two patches for performance examination purposes.
- Patch 1: Original page fault handler abstraction
- Patch 2: Improved page fault handler abstraction
- Examine both with caliper and see what it tells me.
- Set up private cvs repository to keep benchmarking patches
12/10/06
- Spending the next four days on my home network
- Will return to PTI release and benchmarking on Monday. Too involved to do anything else ATM
11/10/06
- Run "official" measurements with caliper on the PTI today.
- Caliper has highlighted some bugs in PTI
- FIRST - Reproduce ICE benchmarks - document on wikipage.
- ON HOLD - after discovering a PTI bug
- Abstract a page fault handler only patch.
- Benchmark with lmbench
- Run caliper measurements on it.
10/10/06
- Spending a few days at home messing with my network facilities.
- Spent couple of hours playing with caliper.
- Ran some tests for a look with caliper on the PTI.
06/10/06
- Skills development: Investigating m4, autoconf
05/10/06
Start plan (DO PRELIM TONIGHT to discuss with Adam tomorrow)
- Spend more time playing with caliper.
- Write a programs to investigate cache pollution. Look at them with caliper.
Spent time reading about caches DONE
04/10/06
Spend time getting to know caliper to assist PTI evalutation DONE - I want to spend more time just looking at the tool later in the week
- Write a couple of simple programs to demonstrate to myself some of the things that caliper can do.
- First - a simple program that has known hotspots that I can tweak. See what caliper reports.
- Second - a program where I can easily tweak expected TLB misses and use caliper to show me.
- Write a couple of simple programs to demonstrate to myself some of the things that caliper can do.
- Spent some time tonight on my LFS installation
03/10/06
- Today - fiddle about with static and dynamic libraries
DONE - to be continued though.
- Continue with my Linux from scratch installation.
DONE Will be ongoing for some time...
- Work on my 3 month plan to be posted on wiki by Friday
Deferred to tomorrow
02/10/06
- Mucking around with various programming tools
29/09/06
- Going to run sister and new baby to airport shortly.
- Start a wiki benchmarking page for the Page Table Interface
- Peter may organise a large machine over the summer
- SPEC benchmarks gcc and gzip
- Postmark
- Use HP-caliper
- Regarding the results below for Page Fault. If the spurious data points are removed and rerun
- sd becomes 0 and the Page Fault will be 2.000 us for No VHPT
- sd becomes 0 and the Page Fault will be 1.000 us for No VHPT
Benchmarking results for ICE talk
***NO PTI***
2.6.17.2 vanilla VHPT OFF, 4 level page table, 16K page
null null open signal signal fork execve /bin/sh
kernel call I/O stat fstat close install handle process process process
------------ ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.56164 2.794 0.711 5.613 0.906 4.058 154.1 787.7 4049.4
s.d. (10 runs) 0.000 0.00058 0.005 0.002 0.023 0.001 0.015 0.0 13.9 30.5
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 55.85 24.28 84.09 42.19 87.35 42.05 123.83 45.02 5806.7 1.715 2.00
s.d. 0.06 0.10 0.52 0.20 0.88 0.22 4.29 0.94 103.4 0.032 1.15
***PTI***
null null open signal signal fork execve /bin/sh
kernel call I/O stat fstat close install handle process process process
------------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.57173 2.794 0.705 5.573 0.907 4.087 157.1 778.5 4090.8
s.d. (10 runs) 0.000 0.00018 0.008 0.000 0.021 0.001 0.010 0.0 20.2 17.1
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 56.07 24.24 84.30 42.16 87.02 42.13 122.16 45.39 5978.1 1.674 1.90
s.d. 0.08 0.11 0.78 0.20 1.08 0.23 4.71 0.14 114.0 0.023 0.32
SUMMARY
2% Slow down on fork
exec and page fault slightly quicker***NO PTI***
2.6.17.2 vanilla VHPT ON, 4 level page table, 16K page
kernel call I/O stat fstat close install handle process process process
------------ ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.56145 2.769 0.712 5.695 0.907 4.090 145.9 769.7 3990.7
s.d. (10 runs) 0.000 0.00022 0.007 0.001 0.017 0.000 0.017 0.0 24.9 16.7
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 56.08 24.41 83.91 42.21 86.54 42.24 119.25 45.50 4917.1 1.618 1.50
s.d. 0.06 0.09 0.75 0.18 1.09 0.22 3.05 0.11 41.4 0.041 1.58
***PTI***
null null open signal signal fork execve /bin/sh
kernel call I/O stat fstat close install handle process process process
---- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
2.6.17.2 0.323 0.57058 2.819 0.705 5.632 0.906 4.111 151.5 780.6 3984.3
s.d. (10 runs) 0.000 6.74948 0.009 0.000 0.029 0.001 0.007 0.0 14.3 16.9
0K 0K 1K 1K 4K 4K 10K 10K Mmap Prot Page
kernel Create Delete Create Delete Create Delete Create Delete Latency Fault Fault
-------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ------
2.6.17.2 56.18 24.53 84.84 42.35 87.29 42.36 119.74 45.55 5024.8 1.576 1.30
s.d. 0.06 0.04 0.27 0.15 0.22 0.19 2.58 0.28 39.9 0.027 0.95
SUMMARY
3.8% fork slow down
exec slightly slower
page fault slightly quicker
28/09/06
- Run Benchmarks
- I have done a run with 5 repeats. I am going to run them again with 10
- Will send "official" results to Peter for tomorrow morning
- Prelim figures: A few percent slower for fork and exec.
27/09/06
- Taking the more abstract content of the poster and putting it in the place of some of the detail in the PTI slides.
- I can't build the latest cvs checkout of the talk. I will look into this.
- Run some benchmarks on latest PTI tomorrow (I only have numbers for 2.6.12)
- Ask Adam how to get the stepwise stuff to work.
- Checked in stuff. Fine tune tomorrow.
26/09/06
- Still tinkering with diagram
- I have been off colour for the last few days. Will hopefully finalise stuff tomorrow.
25/09/06
- Make PTI slides more abstract
- Add notes to PTI slides
- Fix up the diagram
- The required diagram is not in cvs (although all the others seem to be). It is in the final diagram, but it does not exist as an eps or fig.
- Found it.
- Working from home today. Be in all day tomorrow and wednesday.
21/09/06
- Looking at Gelato ICE Talk
- Putting Adams notes in slides. First two slides done (banked in CVS)
- Didn't have time to do the two hybrids.
20/09/06
- Play with Latex
- Done 4 PTI slides. Will do one diagram - not done yet.
- Will check into CVS tonight.
19/09/06
- Picking up Itanium 1 at 5:30 today.
- OMG it is huge. ~70kg. I am going need some assitance to get this thing out of my car and onto a trolly.
- Work on talk - plan diagrams.
- Do the MLPT vrs GPT slides. plan diagrams etc. (At the moment I am doing PTI only). I am bogged down finding out about a bunch of utilities - pspresent, chaksem, texpower, seminar ...
- I will check in the PTI slides ready for Thursday. Will come in and discuss the talk with Adam and Ian on Thursday.
- Look at Ians stuff tonight.
- Checked out and built the talk.
18/09/06
- Work on the PTI/GPT/Superpage talk for Gelato ICE
- Started. There seem to be problems with weill at the moment so I have been unable to check out the talk.
- Installed some of the tools pspresent, texpower. will muck around with them tomorrow.
- Made contact with Chris from Aurema. Will pick it up some time this week.
15/09/06
- Paul still sick. Back Monday
14/09/06
- Paul is ill at the moment
- Will organise to pick up machine from Aurema
11/09/06
Looking at the detail required in for the minute taking. DONE
Fixing up to email in. DONE
- Do late progress report.
I am fiddling with torrone ATM. DEFERRED
- Further aquainting myself with our network.
Have a last go at the abstract. DEFERRED
Work on the abstraction part of the talk. DEFERRED
- Spent the afternoon thinking about VLA's, self mapped page tables and Ian's diagram of a VLA.
- Discussed VLAs in depth with Ian. Discussed what is a cache and what is a TLB.
I confirmed my understanding of the use of the TLB in this regard with Matt Chapman.
- Will do an enhanced diagram to illustrate a VLA. - there is a bit missing in my opinion.
08/09/06
- Ian helped me straigten out my misunderstanding with respect of the a self mapping page table and the virtual linear array.
06/09/06
- Mucking around with CentOS Live cd.
- There is a CentOS 4.4 for i386 but there only appears to be a version 4.3 for ia64.
- I am burning the 4 cds for ia64 version 4.3.
01/09/06
- Read about the demos for the CSE open days.
- Meeting at 4PM regarding this.
- Reading again about the short format VHPT.
- OMG I finally understand it. That didn't take long did it.
31/08/06
- I tried to build the latest LVHPT patches into ~ 2.6.15 I could compile and not boot it (I couldn't find the right config to build it with).
- Rather than ask Ian for a config I decided to retackle ivt.S, the long and short format VPHT etc etc. I started to read Matt Chapmans LVHPT paper (I have advanced a lot since I last tried to read it). I am in the process of doing this. Spent my night revisiting the short format of the page table walker in the IA64 text.
30/08/06
- Work on applying LVHPT So LVHPT (in gelato CVS) is only up to date for 2.6.14. Not surprisingly these patches don't apply to 2.6.17.2.
- I am sure Ian has more recent versions that this tucked away because this stuff is 11 months old and FIND_PTE etc are in defines still not macros.
- Spoke to Ian. He has pointed me to the lastest. It is going to take some effort to get the patches to work with 2.6.17.2.
- Obviously I know how to patch stuff etc. But there are a bunch of manouvres that you must be able to do but don't know how. I am going to put some time into more advanced patching skills to make my life a bit easier.
29/08/06
- Working from home today (I have a dentists appointment in the middle of day)
- I have a list of things to work on given to me by Adam
- Will start working on getting the thing to work will LVHPT (NOT DONE worked on broadening Linux skills instead).
28/08/06
- Get PTI patches to Adam so I can play with kernel modules (send patches)
22/08/06
- Get updated PTI to Adam
- Pull Ians git tree and have a look at his page table walking stuff.
- Got Ian page population module. Got it working. Plan to add some extensions to it next week.
- Adam wants an analysis of pud and pmd levels that contain single pointers through to the next level (where guards will benefit).
21/08/06
- The kernel GPT is being looked up OK. Investigate the liklihood that lookup of user GPT's is failing.
- This is definitely the problem.
18/08/06
- Get synced GPT to Adam by end of day in some form. unless I hit another problem.
- Tonight look at debugfs
17/08/06
- Spend the day on my backtracked GPT patches.
- Not necessary. I found my problem and am getting back on track
16/08/06
- Working on gpt patches.
Still being mystified by a break down. Backtracking here due to frustration
- Mucking around with modules (will talk to Ian but I want to play with modules)
Read about relayfs Now know how to turn it on and what it is for
- Always built monolithic kernels for ia64 in the past. I have created a trivial module and am still in the process of getting it to load.
12/08/06
- Send abstract to Peter
- Working from home today
- Work on GPT in kernel.
- Also look at the framework required to write a kernel module.
- Adam suggested writing a module to gather data page tables use of the address space.
Talk to Ian, discussing this with him on Monday he also said he'd take a look at it - AdamWiggins.
11/08/06
- Write abstract for ICE by the end of the day.
- - approx 10 lines.
- Tidy up the userspace GPT and give to Adam.
10/08/06
- Got Adams original GPT to work with new config.
No anomoloy - just a config problem fortunately
- My synced GPT is breaking down - figure out why
- GPT is reading lookup now. Get stuff tidied up.
09/08/06
- GPT compiled into kernel under full PTI.
- I just need to get it to boot now.
- Look at kerncomp again.
- Back doing further reading on my bash scripting skills.
- Playing around with my understanding of permissions
- It appears to have broken with the change of kernel from rc3 to rc4.
- Is this a coincidence?
- I a switch user to ianw I cannot use sudo - I get pam - authentication failure each time
- On the final set of results the summary.log records the kernel version as rc3 and the git-pull says it pulled the patch for rc4.
08/08/06
- I have been putting Adams GPT under latest PTI and parts of the PTI that need to change have become intermingled with the GPT. It has become a bit of a mess.
- I am going to go back and adjust the IA64 PTI to fit in with Adams stuff
- I have most of the gpt headers compiling in the kernel as intended
- Should have it running by Friday.
07/08/06
- Doctors appointment in gladesville at 11AM then I will be in.
- Working on userspace model etc.
- You will just be able to run a script to update the GPT in the kernel from userspace. ie. develop in userspace, run script and compile for kernel.
- Making a couple of adjustments to PTI to better slip in adams stuff.
- Update wiki for PTI and planning.
04/07/06
- Will do some work tonight. I spent my afternoon chasing around some laptop problems (had to return to the shop).
02/07/06
- Ask Ian to have a look at his cron table to see why the kern comp script
isn't being run. DONE
- I have create, build and lookup compiling without fixing mem allocator. Fix mem alloc and see how it goes. - OK. I see it is not using the quicklists yet.
Starting to read through some of the core GPT code to refresh my memory. STARTED
I can build and lookup the GPT in userspace. DONE
Write an email to Chris Wright about the iterators. DONE
01/07/06
- Continue working on userspace version of new GPT.
- Build a little quicklist allocator for the model.
- Core GPT building in user space - #defining out the allocation still.
- Will start to bring in the interface functions.
- Have another look at the kernel build. Apparently Ian has a cron job for the kerncomp script and it is OK.
- It is an internal problem to do with Ians crontab and permissions.
31/06/06
- Start with looking at why the new GPT isn't booting on tartufi.
- Tested to see if it boots UP - NO.
- It appears to break on the first lookup to me - not related to the dormant race condition.
- Building userspace version.
- Ditch a bunch of unused kernels from home directory before I get shamed.
28/06/06
Go over the IDA64 PTI patches with Adam DONE
- Small changes will be made to PTI to make it fit in with Adams stuff as best possible.
- See if I can't figure out why the assembler lookup is barfing (Adam speak) on tartufi, or most importantly get it to boot by reinserting a variation of the old lookup.
- Probably come up with a new short term plan and some goals.
- Start syncing the GPT under the latest patches.
27/06/06
- Get new GPT patch from Adam and put it in the latest PTI series.
- This will take some time and effort
- Look to see why kernel autobuild isn't running.
- Had a quick look at it. The kerncomp.sh script isn't being run. I don't know why - I would start by looking at the crontab but I haven't got the necessary permissions to investigate any further.
- I don't see any cron job being run by the system-wide crontab. I assume it is being run from ian's crontab.
25/06/06
- Work on final disentanglement
- Now just need to do some cleaning up.
- Working on bash and environment skills.
24/06/06
- Work on IA64 patches
- Working on disentangling the various pgtable-nopud.h pgtable.h abstractions.
22-23/06/06
- Personal development this weekend.
- Reading through the "Cathedral and the Bazaar" - relevant to me because I tend to dither and not want to show anything to people until it is perfect. This greatly reduces my effectiveness. I am still learning to "release early and release often".
- Learning about Linux from Scratch. Plan to follow this through.
- Building networks at home to improve networking skills.
20/06/06
- The collective PTI patch series should be far more stable for a while.
- Do out new prepare GPT patch series - to sync with Adam.
- A stable PTI will enable me to attack performance (which is actually interesting).
19/06/06
Finish IA64 patches and send to Adam today. FINISHED PATCHES
- Do some extra testing prior to sending to Adam
Get PTI to work for x86-64 (so Lee can test it on one of his Opterons). DONE
- Reading.
- Reading about caching.
- Reading about different architectures - about gaining a better understanding of where different architectures fit into the scheme of things.
18/06/06
- I have an appointment with my ENT doctor at 2PM in chatswood.
- Working on IA64 patches for Adam.
17/06/06
Work towards resyncing with Adam High Priority.
- Preparing IA64 patches ASAP so Adam can move again.
- Helped Adam get rid of a legacy assembler hack created by myself.
- Look at PTI performance issues for a small amount of time each day.
- Working on sysadmin skills tonight.
14/07/06
- Think about including HugeTLB in PTI. I left it out initially because it made life easier.
- Work on PTI IA64 patch series.
- Review some theory and get to work on Lee's feedback email.
- Had a look at the external time command.
13/07/06
- Reposted patches to linux-mm. No matter how hard I tried, I could not seem to use the smtp.cse.unsw.edu.au server to send my mails.
10/07/06 - 11/07/06
- Having operation. Indisposed.
09/07/06
Booted final patches on tartufi DONE
Booted final patches on tutti DONE
Running LTP on tartufi. DONE
Sent patches to linux-mm Haven't come thru yet.
08/07/06
Finished splitting up the patches DONE.
- Need to write out the explanation for each patch - then feed.
- Explanations and mails ready.
- Do final test tomorrow and then hit send.
07/07/06
- I managed to get 2.6.17.2 vanilla running on tutti.
- self introduced bug stuffed me up.
- Want to start the split today.
- I will make sure the patch split PTI runs on tutti
- Bug in vmalloc_to_page - here is where to start. This is breaking the PTI on tutti.
The bug is in lookup_page_table (an absolute howler)
PTI now boots on tutti. Will send bug fix to Lee.
- Doing out patch split now.
06/07/06
- Building PTI on tutti.
- PTI crashing tutti - incorrect conclusion. I thought I was booting my kernel when I wasn't. In actualy fact - I still haven't got a 2.6.17.2 kernel booting yet. I have at least got it booting my kernel but it breaking down. Something wrong with the config.
05/07/06
Playing with sendpatchset utility Piece of cake.
- Lee is going to run the patches on an Olympia some time this week.
- Should be interesting. The patch is untested on a NUMA machine. I have only ever run UP or SMP. Hope it doesn't break.
04/07/06
- Working on IA64 dependent interface today.
Send PTI to Lee S. DONE
03/07/06
- Working from home today. I want to finish PTI today and get it to Adam for tomorrow and then straight to Lee.
- Actually I am just going to send PTI to Adam for assessment now. It is 90% done. Then I will send it straight onto Lee when finished.
- More or less finished it. Will send to Adam shortly. Expect to send it to Lee some time tomorrow.
02/07/06
- Getting PTI ready for Adam/Lee.
01/07/06
- Working on PTI for release
- I want to get them to Lee within the next few days.
- Starting to look good. Picked up a number of bugs in fremap.c and filemap_xip.
- I expect to have the PTI ready for Adam on Monday. When Adam has reviewed it, I will send it straight through to Lee. I will then split the patches out pending advice from Adam and Lee
30/06/06
- Building up a next release for the latest kernel.
- Leaving performance for now.
28/06/06
- I have been trying to measure various aspects of the PTI that have slowed down forking. I created a bash and perl script to to run the lmbench fork test a chosen number of times and take the average and standard deviation.
- I am running the fork test with a sample space of 10000. Avg: 143.86 s.d 2.630 ~95% accuracy (2 s.d) 143.86 +- 2x2.63/100 Expected +- 143.86 +- 0.05
- I am now running the fork tests for some individual abstractions that I am interested in. I am studying the cost of an individual
abstraction (pte_alloc). Avg: 144.63 s.d 2.7
- This looks to me like simply moving a function (called critically in memory.c during fork is having a significant impact on performance. Quite alarming to me.
24-27/06/06
- Start rebuilding PTI
- Measuring causes of slow downs.
- I now have very interesting and useful results. I am going to pursue the performance issue on Thursday.
- I will benchmark on x86, as I am suspicious that it something unique to the IA64 architecture causing me problems.
- Measuring causes of slow downs.
23/06/06
- Working on my benchmarking/scripts etc to slash my time wasting.
- I have finished a little set of bash/perl scripts that will save me a huge amount of time.
- bash scripting improving.
- I will be building on these scripts in the near future.
22/06/06
- Mucking around with q-syscollect. I want to do some profiling to see where fork spends its time.
- Figured out more or less how to use it (had a play with it) - will start to make use of this tool in future.
- Work on some scripts for automating benchmarks this afternoon.
- I want my own framework for running benchmarks on tartufi overnight.
21/06/06
- Fix up a couple of known bugs in official patch set.
- Fix up some of the hangover stuff from the name changing.
- Remove function pointer calls from special iterators.
- Demonstrated the principle for copy page range. Now I want to measure the cost of passing functioning pointers before I make any solid decisions about what I do. I am going to produce some patch sets for benchmarking (which will involve x86 and IA64). I just want to get an idea of how costly passing the function pointer is w.r.t my performance problems.
- I also need to do some scripts to automate my benchmarking.
20/06/06
- Continue tidy up of PTI patch set.
Finish fixing the naming scheme for page table files. DONE
Make sure you can at least compile a kernel with the GPT running. DONE
I plan to give Adam a new set of PTI and GPT patches by the end of today. DONE
Stuffed up the patches I sent to Adam. I have been working on this and will resend soon. DONE
Remove passing of function pointers to critical iterators to get back performance. MOVED TO TOMORROW Critical iterators at the moment are earmarked as copy_page_range, unmap_page_range and the free page table funcs.
18/06/06
- I am going to work most of the night as I expect to be up until 4am and not in tomorrow.
- I have run some tests. There is no catastrophic memory leak (as OOM suggests). I need to work out why build_page_table is breaking. - getting a GPT_OVERLAP.
- I have NOW reached the boot prompt. It breaks on very low radices though. (Probably an internal locking issue). I am not going to chase it down now.
- Working on PTI for release late in the week (or thereabouts).
- Fixed mm_struct typedef problem.
- Fixing the file naming problems. Unfortunately it reverberates through the whole patch split.
16/06/06
- Still debugging.
- Build_page_table is breaking causing an erroneous OOM error. It is not really out of memory. I must have stuffed the page table translating it accross somehow.
15/06/06
- Unfortunately I am picking up a number of abstraction errors.
- I will be spending today fixing what I have identified so far. Presumably one or more of these is causing me the problems in the boot process of the GPT.
- During boot, I get "VM: killing process init". I presume this means I am running out of memory (massive memory leak) - which I assume is caused by unmap_page_range.
- Picked up ridiculous errors (introduced by me) into unmap,vmap,copy_page_ranage
and change protection iterators. Must be tired
14/06/06
- I will spend some more time trying to nail some problems on GPT patches.
- Still hunting the problem. I will devote the rest of the day to this.
- Picked up naughty abstraction error in unmap_page_range. Presumably some more to come.
13/06/06
- I have enough for Adam to start working on the GPT, in the kernel. I am going to sort out some patches for him (need to do some tidying).
11/06/06
- I have enough for Adam to start working on the GPT, in the kernel. I am going to sort out some patches for him (need to do some tidying).
10/06/06
- Get GPT going under 2.6.17-rc5.
Compile in GPT implementation DONE- now I need to call it and see how it goes.
Reading assembler lookup DONE
Up to create_user_page_table DONE
- Going OK. Still some work to do.
08/06/06
- Paul is sick at the moment
07/06/06
- Working from home today - I have a bad cold and can't stop sneezing.
06/06/06
- Continue getting GPT patch for Adam
- I have been creating the "prepare for GPT patch".
I will incorporate appropriate elements of this back into the PTI series.
The original PTI series did too much rearranging of stuff confusing the kernel writers. This patch will largely shift back into the PTI for the next PTI version. GPT Prepare patch DONE - Send prepare patch to Adam. It will make it very clear what needs to be done - where he needs to look at etc.
05/06/06
- Concentrate on getting the GPT to run under the old interface.
30/05/06
- Put patches in CVS
- Running benchmarks.
- Feed them tonight.
- Visiting my new born nephew today.
29/05/06
- Benchmarked rc5. 3.5% deterioration in fork. Much better than than the results for 2.6.12.
What was the deterioration for 2.6.12? - AdamWiggins
- No deterioration in mmap latency.
24/05/06
- Working on re-building a full PTI. Going well. Organising patch split as I build it up. Probably feed on weekend.
I have all my patches sorted out up to iterators. I have to actually go ahead and do the split (trivial).
- Enquire to postmaster about patch size limit. 100 000 characters doesn't seem right. Think they mean 10 000.
- Test on i386 tonight - forgot to do my Kconfig for i386. Should do for powerpc as well. before feed.
23/05/06
- Continuing preparation for patch feeding. Very time consuming.
- Also found a several problems with the lmbench sum perl scripts which should be fixed. eg: Failure to calculate s.d. on fork.
- Will start patch split tonight. Do the first three or four patches.
- Patch 0). The big explanation patch. Defer this guy.
- Patch 1). Introduce basic interface.
22/05/06
- I spent my weekend trying to work out how to get the loss in peformance back in the page fault handler. I am pushing on with this today. The interface will change according to the solution to this problem.
- After talking with Adam, I have decided to just go with one of my solutions and work on performance after feeding.
19/05/06
- I cannot resist continuing to try and get to the bottom of PTI performance problems. Benchmarking various ideas at the moment.
- Prepare patches for feeding - architecture dependent and independent.
- Talked to Adam about why I was still losing a little bit on fork. He gave me several things to think about - try them today. This time I am certain all performance overhead will be removed.
- Moving the locking inside the implementation in the page fault handlers to facilitate fine grained locking support into the PTI.
18/05/06
- I patched the user level driver stuff yesterday but it didn't compile Peter has had since fixed this patch set for the latest kernel so I will have a look at it and see where I went wrong.
- As most of the autobuild is now OK I will fix the page fault
handler patches and document it on the page table interface page. LOSS of performance in page fault handler is now 0.5% for fork, nothing for mmap - BIG improvement Actually - I need to work out why fork is still slowed down. I was expecting none at all.
17/05/06
Patch user driver stuff and put in CVS TODO - put in CVS
Fix up the configs for the i2 - uniprocessor and preemptible DONE
- Change firmware for tartufi and cassata
I have deferred this to the weekend. I have asked Ian not to do it. I realise it is a five minute job - but I want to read about EFI etc first and do
- Spent some time this afternoon talking with Adam about just why they do pass pmds around the page faulting functions and relook them up. (Due to highmem on i386)
16/05/06
- Experimented with the build scripts tonight. I have been migrating the scripts across to work in my own directories - just to have a fiddle with them. I don't have the permissions to work in situ. Need to fix this. I think we should shift the kernel autobuilding out of ians home directory into a general gelato directory.
- I am 100% sure I know exactly what I am doing and should be able to sort it no problems. (OK - I had a minor mishap - grrr cut and paste error)
- Making good progress - I am cloning/fetching the latest kernel correctly. I am rebuilding them back up, it is the easiest way to learn bash. I haven't got it to build successfully yet - but I want to go to bed.
- TODO: Update the wiki page on the kernel autobuild
- I was running my experimental scripts (partially built) and my output was written to the real scripts output. (I cut and paste - forgot to cut). The real scripts have now been run and new output is being generated.
- LESSON: Should not have experimented with scripts on eisbock. Too easy to make a mistake.
I might add my pti patches to the autobuild - This is definately worthwhile (AdamWiggins)! (OK PaulDavies)
- Investigation why user level drivers patch doesn't apply in autobuild.
System calls have been added into the ia64 syscall patch. Easy to fix.
Similarly other syscalls have been added to i386. Easy to fix.
- Investigate why two kernels failed to build. Work out the config conflicts.
- Have a look at why vhpt-sim patches don't apply. Get from lists.
15/05/06
- I will be a little late as I have a doctors appointment at 9:20 in Chatswood.
- I am going to look into loss of performance in the page fault functions today
- I have a 3% deterioration in fork due to page fault abstractions alone. I have been aware of this for some time and have a solution in mind. I will see how it goes.
- Start on the problem of fixing the kernel autobuilding. Finding out a little about git and cogito. The version of git has now been upgraded to the latest version.
07/05/06
CeBit tomorrow.
Add a wiki page to explain how to get a kernel up and going - PaulsKernelBuild
- Reading about vNUMA and linux on linux.
06/05/06
- Reading about basic locking primatives.
- Reading about memory barriers.
- Examined the reader/writer concurrency problem. Wrote a program using the pthread libraries to demonstrate and understand the problem in my mind. - still in progress but I expect to finish it tomorrow.
05/05/06
- Working on my in depth understanding of the page table in general
- There is a lot of discussion on linux-mm about a possible page table walking problem as a result of calling free_pg_tables. It is providing a very informative discussion on the page tables in general.
- Learn about memchecker, PIN, memtoy and caliper.
04/05/06
- Read about Nick Piggins lockless radix tree.
03/05/06
- Have doctors appointment at 9. Will be in after that.
02/05/06
- Started to look at the finer details of some of the locking in VM.
- Working on documentation
01/05/06
- Move arch independent interface to 2.6.17-rc3. DONE
28/04/06
- Sent the architecture independent interface to Adam for latest kernel so he can look at it on the weekend.
27/04/06
- Very happy with PTI (architecture independent) - will post to CVS tomorrow.
- I have two custom dual iterators (copy_page_range and move_page_tables).
- I have five iterations that call a general read iterator
- I have two customised read iterations.
- I have one general build iterator - I am going to leave remap_pfn_range individually abstracted as judging from linux-mm this is about to change.
- I will get the interface documented on the page_table_interface over the next few days.
- I then plan to get my architecture dependent IA64 patch ready and documented by the time Peter gets back.
- EXPECT TO HAVE PTI ready for feeding (pending code review by Peter) by Monday.
26/04/06
- I have 2.6.17 PTI abstracted with all iterators abstracted individually. I will now change it to a handful of specialized iterators and those that call the general iterators.
- Abstracting them all individually gives me a good feel for the final PTI.
25/04/06
- Work on page table interface page documentation
- Finish the architecture indpendent PTI patch.
24/04/06
- Starting to work on the write up on the page table interface page as I am relatively happy with the PTI patch progress now.
- Will send patches to Adam soon.
18/04/06
- Working on PTI for 2.6.17-rc1. Aiming to have it ready by the end of the week.
- working from home this week (no one is there). I will go in and see Adam when we arrange a time though.
- Mucking around with testing aspects of the interface I have yet to test. eg: fiddling with swapon and swapoff. Need to fix config - swap space signature problem. Need to run mkswap.
17/04/06
- I have decided that trying to share the code for the pgd pud and pmd iterations and specialise the pte iteration isn't the way to go. I have proven to myself I can make it work but I don't think it is the right thing to do. It is far simpler to duplication code - and I want the PTI kept simple.
14/04/06
- Overhaul of PTI for 2.6.17 underway. Reworking the iterators at the moment. It is going well.
- The iterators will now share the pgd, pud and pmd iterations but the pte level can be specialised. It is the cleanest solution to the customisation of iterator problem.
- My new MLPT copy iterator is benchmarking very well on ia64 but needs to be tested on x86.
13/04/06
- Start preparing PTI for 2.6.17 for code review.
I am in Queensland from 7/06/06 to 13/06/04 and consequently unavailable.
06/04/06
- Mental note to self. Having group write enabled on .ssh directory will stop sending packet for ssh key authentication.
05/04/06
- I am going to change the GPT coverage to the minimum number of bits required for each of the 4K, 8K, 16K and 64K pages. I currently cover the whole 64 bits. I will tidy up the GPT user space model while I do this. I plan on banking the page table test framework with model GPT and MLPT into CVS in the near future.
- FRAMEWORK /framework/gpt/ -- all gpt source *.c and *.h in this directory. These files
- will be able to lifted straight from the framework and dropped into the gpt directory in the kernel.
- against MLPT etc
04/04/06
- Working on my building a userspace test framework for the GPT/MLPT. This will enable me to benchmark the GPT vrs MLPT nice and easily in userspace.
- I am still working on my command line utility to populate the address space with controlled data sets.
- I now have a utility to create decent data sets. I will do some cleaning up and further enhancement of this in the near future.
- VERIFIED today that my read iterator is performing as expected over random data (where as the crap iterator doesn't). I need to get onto the threaded trie version to benchmark against my stack iterator.
- Tomorrow - back on my GPT model, possibly incorporate the MLPT into my framework.
03/04/06
- Working on user space model of GPT.
- Writing a utility to generate different kinds of test data for speed testing the iterators.
- Early testing of iterators show they are not as quick as I thought they would be. However, I am quite certain I know why. Before I go ahead and fix it though, I want a utility to generate different kinds of GPT key/address space layouts. Then I will directly compare the GPT against the MLPT.
31/03/06
- Having my first look at Adams new GPT
- There are a great many functions in many directories. It will time to convert it to 64 bit and get it running in user space.
28/03/06
- Fixing CV
27/03/06
- Get a machine and load Linux (DONE).
- Configure machine (reading about LDAP)
- Fix the broken links for aim7 on wiki.
26/03/06
- Did CV for Emilia (will refine further tomorrow)
- Did Job Description for Peter (will refine further tomorrow)
08/03/06
- Added a single patch to push to 2.6.12-rc2 for LVHPT, PTI and GPT.
- On holidays for next 10 days or so.
02/03/06
- I had two bugs in the read iterator. 1) I had it in my head that an invalid was zeroed out. WRONG. This stuffed up my guard and ultimately my coverage.
- My stack was too big (YES I was warned). I didn't heed it. It made it look like there was something wrong with my fundamental method of iteration (NOT THE CASE).
- So - I have a working read iterator in 2.6.12-rc2. This will easily roll forward to 2.6.15-rc7. I have a lot of faith in the PTI so I went back to 2.6.12-rc2.
- Now I can get back on track.
01/03/06
- I am fixing up the 2.6.12-rc2 patch to go in CVS. I think this is important. It will include a fixed up version of the GPT that should perform reasonably.
- I have a single patch with cleaned up GPT. I will add proper read and build iterator tomorrow.
27/02/06
- I am going to look at Adams read iterator (Later - clean first).
- Start cleaning up my files etc.
- CVS is a mess. I will toss out most of it. I am going to create a single patch of the GPT running rough under 2.6.12-rc2. If nothing else this is something running that could be used as a starting point for another person.
18/02/06
- After seeing Gernot's email I abandoned coding and am writing a quick report (~5 pages only).
- I am working on building a linked list between the leaves of the trie. Essentially the overhead of "iteration" is shifted into the build process as a single iteration to find the previous node is required (at least on a one off build).
- This involves hacking Adams core libraries (which I was told was a major NO NO).
- I wrote my current read iterator acting on Adams advice (which I trust) - and at the very least Adam's trie. It is no longer Adam's trie model if you do this kind of thing.
- Doing this kind of thing causes the trie to diverge from Adams and it will make taking in Adams update time consuming and prone to introducing errors.
- Other than that, if you are prepared to wear the bigger node and the overhead shift is desirable, its a very funky idea.
17/02/06
- I have doubled the size of a node from 2 unsigned longs to 4 unsigned longs and am working on a "threaded" read iterator. It will be worth benchmarking the stack iterator versus this. I am doing this instead of playing with copy for today.
- The idea being - iteration will involve traversing a linked list, (and the cost of finding the start node). The iteration cost is simply shifted into the build process.
16/02/06
- Happy with build iterator. Working on copy now.
- The build iterator is fine (will send it to Adam and Peter before the Friday group meeting). I reworked it when paulaner re-appeared again tonight (breaking nightly is apparently routinely acceptable). However, while it will improve performance slightly with small radices, to me it seems pointless for larger radices (and real life).
- Additionally, the copy iterator is far more complex than the build. I know what is in the range of the build which makes it very simple (despite the time it has taken me to get it to work). So far, all I have done is abstract, writing is a whole different kettle of fish. I need to get into the groove. Thats the truth.
- I have built the build iterator to adhear with Adams advice with regards to the build iterator - and yes, GOOD advice. The problem is, the build iterator, in itself was never really a problem iterator in the first place. Read and copy are were the problems.
- I will start to build the copy iterator, but I will play it by ear and see how it pans out. My build is going to have to change to fit in with the read for an efficient read/build iterator.
- READ
struct gpt_read_stack {
GPTKey position; /* Position in trie */
GPTNode *node; /* Node */
int pos; /* Position in level */
Log2Size level_size; /* Level size */
int pop; /* To pop or not to pop */
};
static inline int page_table_read_iterator(struct mm_struct *mm,
unsigned long addr, unsigned long end, pte_callback_t func, void *args)
{
struct gpt_read_stack stack[GPT_READ_STACK];
struct gpt_read_stack *sp=&stack[1];
GPTKey temp, guard, level;
GPTKey start_coverage, end_coverage, coverage;
GPTKey key, end_key;
unsigned long address;
int err=0;
(sp-1)->pop = 1;
sp->pos = 0;
sp->pop = 0;
sp->level_size = 0;
sp->node = &mm->node;
gptKeySet(0,0, &sp->position);
gptKeySet(extract_key(addr), GPT_KEY_LENGTH_MAX, &key);
gptKeySet(extract_key(end), GPT_KEY_LENGTH_MAX, &end_key);
spin_lock(mm->page_table_lock);
for(;;) {
/* pop the stack - prevent underflow - */
while(sp->pop) if(--sp==stack) return 0;
/* Get the start and end coverage of our position in the trie */
gptKeySet(sp->node->internal.guard_value, sp->node->internal.guard_length, &coverage);
gptKeysMergeLSB(sp->position, &coverage);
start_coverage = coverage;
start_coverage.value <<= GPT_KEY_LENGTH_MAX - coverage.length;
start_coverage.length = GPT_KEY_LENGTH_MAX;
end_coverage = start_coverage;
end_coverage.value += ((1UL<<(GPT_KEY_LENGTH_MAX - coverage.length)) - 1);
/* Ensure we go to the right initialisation point */
if(key.value <= start_coverage.value)
key = start_coverage;
/* Iterated past our region */
if(key.value >= end_key.value) return 0;
switch(sp->node->raw.type) {
case GPT_NODE_TYPE_LEAF:
gptKeySet(sp->node->leaf.guard_value,
sp->node->leaf.guard_length, &temp);
gptKeysMergeLSB(sp->position, &temp);
address = get_real_address(temp.value);
if(address>=addr)
if(err = func(mm,&(sp->node)->leaf.payload, address, args)) return err;
sp->position.value++;
sp->node++;
sp->pos++;
sp->pop = 0;
break;
case GPT_NODE_TYPE_INTERNAL:
if((key.value >= start_coverage.value) && (key.value <= end_coverage.value)) {
sp++; /* PUSH STACK */
gptKeySet((sp-1)->node->internal.guard_value,
(sp-1)->node->internal.guard_length, &guard);
gptKeySet(0x0, (sp-1)->node->internal.level_log2size, &level);
merge_3keys((sp-1)->position, guard, level, &sp->position);
sp->pos = 0;
sp->node = (GPTNode *)__va((sp-1)->node->internal.level_ptr);
sp->level_size = ((1<<((sp-1)->node->internal.level_log2size))-1);
sp->pop = 0;
(sp-1)->position.value++;
(sp-1)->pos++;
(sp-1)->node++;
if((sp-1)->pos > (sp-1)->level_size) (sp-1)->pop = 1;
} else {
sp->position.value++;
sp->node++;
sp->pos++;
sp->pop=0;
if(sp->pos>sp->level_size) sp->pop = 1;
}
break;
case GPT_NODE_TYPE_INVALID:
sp->position.value++;
sp->node++;
sp->pos++;
sp->pop = 0;
if(sp->pos>sp->level_size) sp->pop = 1;
break;
}
}
spin_unlock(mm->page_table_lock);
return err;
}
08/02/06
- re-wrote the read iterator to get rid of ridiculous unnecessary initialisation routine.
- Plan to implement Adams advice for build iterator - probably Thursday. I have a few little issues I want to tidy up tomorrow.
06/02/06
- Its been a tough week, but I am starting to shape up with the read, build and copy iterator (in user space).
- I don't plan on coming in at all this week. I am working on iterators from home until the job is done (unless I hit a brick wall). I am going to give this week and next week everything to get the RESULTS I am looking for.
31/01/06
- I have a couple of copy iterators for benchmarking well underway.
- The latest (and presumably the quickest) will be copying the trie straight into a trie with one level. It should be ready tomorrow.
- I wanted to produce an efficient iterator that duplicated the trie structure but I made an oversight and I have to go back and rework it. Ultimately I don't think I can get this one to be nearly as efficent as I was hoping.
26/01/06
- I have put the read iterator in the kernel but I have overlooked a case somewhere because it breaks down. So - going back to debug iterator in user space.
- OK - debugged the iterator - now it works in the kernel . I have a few different ideas for the copy iterators. I think I can make it quicker than what Adam has in mind, but we will see. I will be benchmarking them off against each other during next week.
25/01/06
- I am now working on making variouis aspects of the iterator code look professional. I am effectively unravelling aspects of Adam's GPT and putting them directly in the iterator code.
25/01/06
- I have an efficient dual iterator copy subtrie working. I can copy an entire page table as efficiently as an MLPT. I have to deal with copying only sections of page tables using the efficient subtrie dual iterator. However I need to maintain Adams trie structuring rules, so only sections will be able to be copied with the subtrie iterator depending on the structure of the trie being copied.
- Waiting for a refridgerator between 9-5 so working from old home so I can get to new home ASAP (no internet yet at new home).
24/01/06
- Still refining copy_page_range iterator. Going fine. Abandoned shallow copy concept for now. Working on a very simple deep copy of the page table that will match the MLPT.
18/01/06
- Read iterator is finished (I haven't tested it in the kernel).
- I am going to start the build iterator today. This is far trickier.
- At this stage I have decided to allocate a big slab effectively making it a variable radix page table.
- The Read iterator is testing quite nicely with a sample variable radix page table I have just built.
- I think I know how to tackle the build iterator. I am still thinking on the copy iterator.
17/01/06
- Spent afternoon working on iterator. It is dramatically improved but I am still testing.
- It is far simpler and visits each node just once.
11/01/06
- I am now working on tartufi (building kernels etc) with the GPT running in the latest kernel.
10/01/06
- Booting off the GPT kernel page table with node moved up into mm_struct. User page tables not booting yet.
- The OS now boots with the GPT with the root node contained in the mm_struct. At the moment this node needs to be in the right place in the mm_struct. I will fix this.
- A couple of small changes to the PTI resulted from the removal of the redundant level from the GPT.
- I have done a really dodgy temporary iterator for the GPT for mremap.
09/01/06
- Worked on PTI during the day. Well in hand.
- Started to work on GPT tonight. I am shifting the base node up into the mm_struct to remove the reduntant level. Doing work on assembler lookup.
07/01/06
- Passes LTP with the GPT now. Still a little to do to finalise interface though.
05/01/06
- I have reached the boot prompt running the GPT for the page table interface for 2.6.15-rc7.
30/12/05
- GPT is not booting under the latest interface at the moment. Its breaking down very early in the boot. There looks to be bootmem allocation problem which I am looking into into.
23/12/05
- I have done the first pass of the new page table interface. I will start to go through for anything I have missed. I will go through with a fine tooth comb some time before new year.
- I need to break up into small patch series for feeding as well.
- I will not be feeding the interface onto linux-mm until I have verified that the GPT is operational under it first. I don't expect this to be an issue unless I have stuffed up the interface somewhere. THEN I will start to clean up the GPT.
22/12/05
- Unfortunately, I have had to add several new iterators to the page table iterface, which marrs its over all beauty.
21/12/05
- Making good progress. Expect to finish PTI for 2.6.15-rc5 by Friday
05/12/05
- I Lost my workstation to Ravi. So I seem to be waiting for a machine again.
- I have spent the day showing ravi how to build kernels, patch stuff, use mercurial etc...
01/12/05
- Working on tearing down the page tables - the last part of the interface.
- All work is now being done for the latest kernel.
29/11/05
- Got workstation up and running - day and a half delay.
- Thinking about repacking the node.
- Thinking about overall summer strategy.
28/11/05
- Will be in today. Just got to register car first.
- Develop summer strategy.
13/10/05
- I will spend today on the report
- Tomorrow doing a code clean to ready it for evaluation.
05/10/05
- clear page range looking good in the kernel - I still have quite a bit of testing to go yet, but it looks good. I want to be certain things are being cleaned up OK.
04/10/05
- Read iterator looks good in user space
- clear page range looking good in user space
- Will be running test today for different radices etc before the real test - the kernel.
23/09/05
- Finalising a decent read iterator.
- The OS seems to run fine for high radices (1024 etc) with the long format VHPT, but breaks on low radices (2 or 4). Weird!!! Something is not right. I have run out of ideas for the moment. At this stage - I am blaming the lvhpt patches. I am not convinced they work properly for 2.6.12-rc2. I must be wrong but I need to work out why.
- On low radices the GPT itself breaks (it asserts) for long format VHPT only. I just don't get it at the moment. There is clearly nothing wrong with the GPT itself.
21/09/05
- Having problems with VHPT walker - I think I know why.
#define tmp r20 /* tmp val to work out key */
#define pnode r19 /* pointer to node */
#define node r20 /* lower node word */
#define key r21 /* lookup key */
#define multiplier r22
#define size r23 /* size of node level */
#define cmp_value r24 /* cmp val with guard */
#define guard r25 /* guard */
#define length r26 /* guard length */
#define type r27 /* node type */
#define shift r28 /* justify guard shift */
#define inc r18
#define FIND_PTE(va, ppte, fail, ok, L1, L2) \
rsm psr.dt; \
mov r19=IA64_KR(PT_BASE); \
shr.u tmp = va,60; \
;; \
shr.u r17=r16,61; \
shl tmp = tmp,60; \
dep.z key = va,4,56; \
;; \
cmp.eq p6,p0=5,r17; \
add key = key, tmp; \
;; \
srlz.d; \
LOAD_PHYSICAL(p6, r19, swapper_pg_dir); \
shr.u key = key, (PAGE_SHIFT-10); /* key set */ \
;; \
.L1: \
ld8 node = [pnode]; /* get lower node word */ \
;; \
extr.u type = node,0,2; /* node type */ \
extr.u length = node,8,6; /* guard length */ \
extr.u guard = node,14,50; /* guard */ \
extr.u size = node,2,6; /* level size */ \
;; \
sub shift = 64,length; \
sub multiplier = 64, size; \
;; \
shl guard = guard,shift; /* guard ready for compare */ \
shr cmp_value = key,shift; \
;; \
shl cmp_value = cmp_value, shift; /* compare value */ \
shl key = key,length; /* Strip guard from key */ \
;; \
shr.u inc = key,multiplier; \
cmp.ne p6, p0 = 1,type; /* If leaf or invalid break */ \
(p6) br.cond.dptk .L2; \
;; \
cmp.ne p6, p0 = cmp_value,guard; \
(p6) br.cond.dptk .L2; /* If guards equal continue */ \
add pnode = 8, pnode; /* increment node ptr */ \
shl key = key,size; /* strip level from va */ \
shl inc = inc,4; \
;; \
ld8 pnode = [pnode]; /* Get pointer to next level */ \
;; \
cmp.eq p6,p7 = pnode,r0; \
(p6) br.cond.dptk .L2; /* lookup failure */ \
;; \
add pnode = pnode, inc; /* point to next node */ \
br .L1; /* loop */ \
;; \
.L2: \
cmp.ne p6, p7 = 2,type; \
add pnode = 8, pnode; \
;; \
(p7) cmp.ne p6, p7 = cmp_value,guard; \
mov ppte = pnode; \
;; \
20/09/05
- It now boots with the long format walker switched on.
- I will now try out my new lookup.
19/09/05
- Fixed automounter problem.
- I have a patch set to clean the interface which is applied after the LVHPT patches. Then I push the GPT patches. I can chose either of the page tables and only the relevant one is compiled in. They both boot OK. The GPT can of course have a choice of fixed radix.
- I still have a number of issues to sort. Main issue: tinkering with pti-interface I sometimes get a message about a gam_server and unaligned access. I need to fix this.
- I also have to stick in my new lookup (I have simply been running with my reliable lookup til now). I have done a lot of work on a new one and will be testing this soon.
Bad flu again. Hope to get back to work soon.
06/09/05
- Rewriting the assembler lookup - I can do a much better job.
- I am currently putting everthing together.
02/09/05
- BUG FIXING. I had a problem with Bad swap pages but I think I have resolved it.
31/07/05
- FIXED the problem with why I could only get the GPT to work with a fixed radix of 2. I had overlooked a case in the assembler lookup - which has held me up for several days.
- NOW i can get back to the iterators.
26/07/05
- I am working on real iterators.
- I am also going to make a change to how I have packed the node which means rejigging some assembler.
25/07/05
- Seems to pass LTP tests. Start working on real iterators tomorrow to make it bench OK.
24/07/05
- Working on gpt patches where you can vary the radix at compile time. Work towards getting some benchmarks before Sunday.
17/07/05
- Temporary hold up. Bad chest cold/flu. Hope to get back working on GPT in a few days.
16/07/05
- I am changing course. I am going to go through the GPT in detail and work out exactly what needs to be done to implement level compression.
- With regards to memory allocation, I have decided to move Adams slab allocator (which cannot be resized YET) into the kernel. I will use one massive slab to supply the user page tables. I will use a different slab for the kernel page table. I am using this approach because it is convenient at this moment - if I want to get on with level compression. I have other memory allocation schemes in mind. I think I can get this to work for me very quickly which is why I am going to use it. A really large slab should dodge the resizing problems in the short term.
12/07/05
- Investigating memory allocation approaches.
- Created my dirty gpt boot prompt patch.
08/07/05
- Back from Byron bay - start work again.
- Reached copy page range - fortunately nothing wrong with GPT, it was just me.
- Breaking at automounter now.
29/06/05
- I am now booting all the way through the kernel page table up to the call to vmalloc. I am about to make vmalloc go.
- I have now rewritten a lot of what I lost due to the my failure to backup and am starting to make progress towards the boot prompt.
- I am still pulling a stroke to boot through the kernel page table which I will have to fix shortly.
- Backing up is a trade off between the effort to backup and rewriting what you haven't got around to backing up.
28/06/05
- I was lagging behind with my backups and lost a couple of days work. I hope to be back to where I was by late tonight.
- having a quick look at head.S
27/06/05
- Solved my ar.k7 register problem.
- Found why I am breaking down during boot (the immediate reason anyway). I will attempt to remedy it tomorrow.
- Fixing up tartufi after reimage.
- It will take me until the end of tomorrow to get back to where I was after I failed to back up a couple of things.
23/06/05
- I am BOOTING off the GPT. It is reading my assembler lookup in nested dtlb miss. T
- At this stage I only have the kernel page table built from bootmem but I will be moving further towards a boot prompt in the coming days.
20/06/05
- ZEROing in on problem. It is not with the assembler!!! Hoping for a break through later today. Things are very complex at the moment.
- I am trying to boot off the GPT. The kernel GPT is being built from bootmem correctly. I get a nested dtlb miss and the GPT assembler lookup is being called.
- It is erroneously failing the lookup and a page fault results. The page table entry is there. So I will be fiddling with assembler all day.
- I have a several little assembler ditties to write - which will reveal some info for me.
19/06/05
- I am going to start putting my assembler lookup into kernel space and lookup my kernel page table. I want to run some more little tests/experiments. I expect my assembler fiddling will take a few more days yet.
- After this, I will attempt to get some way into the boot process off the GPT.
18/06/05
- Been up quite a few blind alleys recently.
- Building the memory map is now working
16/06/05
- To much involved in reworking Adams allocator to work with the bootmem. I am using a different solution.
15/06/05
- I can now build a page table from different memory sources.
- I have been slowed up by the kernel page table. A very small section is built from bootmem memory. I have been thinking about the ramifications of building the GPT (and deleting from) different memory sources. It must have the ability to effect level compression.
- I am going to carve up the bootmem memory with Adams slab allocator.
14/06/05
- Building a run patch series to add after the clean interface patch series This is a patch which is intended to make the machine boot off the GPT.
- Step 1) eliminate every interface call that does not effect the boot process.
- Step 2) put GPT back in
- Put code to switch between the MLPT and the GPT completely.
13/06/05
- Finalising the GPT under the interface today.
- Tomorrow, I will put it in the patch and hold my breath. Make that the weekend at the rate I am going. However, I am very hopeful I can get it to work before the next progress report. Fingers crossed.
- TARTUFI IS back... and it is time to see if it runs.
Assembler
- The lookup below is the handcrafted assembler lookup for the GPT.
- I will be reworking the GPT interface next week when I get tartufi back. I left a lot of stuff on tartufi (I'm a naughty boy) and I really need it. I'm working with a dated GPT (Adam has changed the interface quite a lot form what I am working with).
- The lookup does seem to be working - I can build a tree with a 1000 random values - look up the thousand values - also make sure I can look up a thousand values I know aren't there and it passes that.
#define node r20 /* lower node word */
#define key r21 /* lookup key */
#define pnode r22 /* pointer to node */
#define size r23 /* size of node level */
#define node_guard r24 /* node guard */
#define va_guard r25 /* key guard */
#define length r26 /* node guard length */
#define type r27 /* node type */
#define shift r28 /* justify guard shift */
#define multiplier va_guard /* another shift */
#define inc_node node_guard /* point to right node */
/* in level */
#define retval r29 /* return value */
#define result r8
#define LOOKUP_GPT(ptr, addr) \
mov pnode = ptr; /* point to root node */ \
mov key = addr; \
mov retval = r0; /* default return NULL */ \
;; \
shl key = key, 14; /* left justify 50 bit key */ \
.L1: \
ld8 node = [pnode]; /* get lower node word */ \
;; \
extr.u type = node,0,2; /* get node type */ \
extr.u length = node,8,6; /* length of node guard */ \
;; \
sub shift = 64,length; \
extr.u va_guard = node,14,50; \
;; \
shl va_guard = va_guard,shift; /* va guard ready */ \
shr node_guard = key,shift; /* for compare */ \
;; \
shl node_guard = node_guard, shift; /* node guard */ \
extr.u size = node,2,6; /* ready for compare */ \
;; \
cmp.eq p6, p0 = 2,type; /* If leaf break */ \
(p6) br.cond.dptk .L2; \
;; \
cmp.ne p6, p0 = node_guard,va_guard; \
(p6) br.cond.dptk .L2; /* If guards equal continue */ \
;; \
shl key = key,length; /* Strip guard from key */ \
sub multiplier = 64, size; \
add pnode = 8, pnode; /* increment node ptr */ \
;; \
shr.u inc_node = key,multiplier; \
ld8 pnode = [pnode]; /* Get pointer to next level */ \
;; \
cmp.eq p6,p0 = pnode,r0; \
(p6) br.cond.dptk .L3; /* Lookup failure - break */ \
;; \
shl key = key,size; /* Strip level from VA */ \
shl inc_node = inc_node,4; \
;; \
add pnode = pnode, inc_node;/* Point to next node */ \
br .L1; /* loop */ \
;; \
.L2: \
cmp.ne p6, p0 = 2,type; \
(p6) br.cond.dptk .L3; /* Not a Leaf - lookup fail */ \
;; \
cmp.ne p6, p0 = node_guard,va_guard; \
(p6) br.cond.dptk .L3; /* If guards OK - continue */ \
;; \
mov retval = pnode; /* Return pointer to pte */ \
;; \
add retval = 8, retval; \
;; \
.L3: mov result = retval; \
;;
08/06/05
- Slight problem with assembler lookup. I can fill the GPT with hundreds of random values and look them all up OK. When I test it to look up values that I know aren't there I sometimes get a segmentation fault. I have traced the problem and it is a bit strange.
- I have made a number of changes to the below code - but I won't put the new version up until I have fixed the problem.
- I will #define the names of all the releveant registers and turn it into a macro (maybe)
- The GPT has a node packed into 2 64 bit words - using 50 bit keys.
- Note to myself - cmp does to a 64 bit compare - stop making my life hard paul
- Note to myself - if you don't need the second predicate on a compare use p0
- FIXED segmentation fault problem on lookup.
07/06/05
- Below is the user space handcrafted assembler lookup. Its working for the first couple of really small trees I have tried.
- Still under test BIG TIME - the gist of it is below. This is just pulled straight out of the userspace function I have been building it in. I may yet decide to change the node encoding which will alter the assembler lookup slightly.
- NB - look for replacement for cmp4 - comparing r24 and r25 - this is a 32 bit compare not 64. Doing two cmps in one bundle looks dodgy.
- I hope to have it ready for transfer to user space by Friday.
- I have reworked the below code to do the 64 bit comparisons in two steps. It isn't as nice though. but it now actually works.
shl r21 = r21,14
.L1: /* Start for */
ld8 r20 = [r22] /* r0 - Lower word of node */
;;
extr.u r27 = r20,0,2 /* r7 - Node type */
extr.u r26 = r20,8,6 /* r6 - Length of node guard */
;;
sub r29 = 64,r26 /* r9 - Bit shift value */
extr.u r25 = r20,14,50 /* r5 prepare */
;;
shl r25 = r25,r29 /* r5 - Node guard */
shr r24 = r21,r29
;;
shl r24 = r24, r29 /* r4 - VA guard */
extr.u r23 = r20,2,6 /* r3 - Level size (post if) */
mov r28 = 8 /* inc val for ptr (post if) */
;;
cmp.eq p6, p7 = 2,r27
(p6) br.cond.dptk .L2 /* If r7 is a LEAF goto L2 */
;;
cmp4.ne p8, p9 = r24,r25
(p8) br.cond.dptk .L2 /* If r4!=r5 goto L2 */
;;
shl r21 = r21,r26 /* Strip guard from VA */
sub r25 = 64, r23
add r22 = r22, r28 /* Inc pointer */
;;
shr.u r24 = r21,r25 /* r4 - Level index from VA */
ld8 r22 = [r22] /* Get contents of pointer */
;;
shl r21 = r21,r23 /* Strip level from VA */
shl r24= r24,4 /* Multiply r24 by node size */
;;
add r22 = r22, r24 /* Point to next node */
br .L1 /* end for */
;;
.L2:
mov r20 = r0 /* Prepare for NULL return */
;;
cmp.ne p6, p7 = 2,r27 (p6) br.cond.dptk .L3 /* If r7 is not a LEAF goto L3 */
;;
cmp4.ne p8, p9 = r24,r25
(p8) br.cond.dptk .L3 /* If r4!=r5 goto L3 */
;;
mov r20 = r22 /* Return pointer to pte */
;;
.L3:
mov r8 = r20
29/06/05
- Trying to get to the bottom of nested dtlb miss walking the page table in physical mode. I am assuming that the GPT must also walk the GPT in physical mode in nested dtlb miss???
- Must get to the bottom of whether or not the GPT must pull the same stunt with pointers that the MLPT does. I am pretty certain the answer is yes. Will confirm
28/06/05
- Working on assembler lookup.
27/06/05
Writing the simplest GPT lookup I can and compiling it into assembler
to see what I can do with it.- Adam changed gptTrieLookup interface for me.
- Confirmed with Adam that temporary iterators are the best way forward to get it working.
24/07/06
- Dropping it in kernel
22/06/05
- I am going to put together a read and build iterator for my testing. (Not the sophisticated version using stack - inefficient junky starting point iterators).
- I don't expect to start putting the GPT into the kernel until Saturday.
The test: insert a million entries:
do 1000 timesX = random value from key Pool.
Y = random value from key Pool.
delete the X key;
insert the Y key;
To test if the trie is holding the correct values. I will use the
iterator to output all trie values and compare with test rig's array.
- Think briefly about an efficient iterator
21/06/05
- I am just finishing putting some tests together that will run the gpt inserting random data sets. I will run the tests for a few hours.
- I will start whipping up another couple of tests while it is running.
- At the moment it is actually failing my testing. It may however yet turn out to be a bug in my testing code. I will keep working on my test code today. Hope to have some strong results by late today/early hours of tomorrow morning.
- I want to come up with some strong test code that I can run each time Adam produces a new iteration of GPT. Hopefully the problem is only with the test code.
- Found a problem with my test code. WOW - the testing is looking great so far. I can pump in 500000 random addresses no problem. I will probably continue to test for a few days yet.
20/06/05
- I am going to suspend the process of putting it in the kernel and doing temporary iterators until wednesday.
- I am going back to user space and I will be running tests with random values to make sure it will hold up.
- Then I will build temporary iterators.
- I will put the 64 bit version with temporary iterators and a temporary memory allocation solution into the kernel next to the MLPT and see if I can test for correctness.
- When I have tested the GPT side by side with the MLPT for correctness - I will try to sort the assembler lookup.
- Then I will be able to run the GPT for real.
18/06/05
- I have a 64 bit version of the page table running in user space.
- I am currently fitting it under the interface. Create, Destroy, Insert and Lookup done. Still need to do the iterators.
- A few issues are coming to the fore which will need to be discussed with Adam. I am doing an UGLY version - I want to see if I can get the OS to run by the end of the week with the fixed radix GPT.
- When the ugly version is done - I will sit down with Adam and see what he will and won't do to accomodate a more elegant solution (elegant for LINUX that is). I realise Linux is not the nos 1 priority.
- When I have the GPT running I would like to run though some of the intricies of the GPT with Adam and start the level compression.
17/06/05
- working on GPT
16/06/05
- Now I have a working GPT with makefiles, I will play with it. I have some 64 bit data structures from the summer session so I will be think about how to integrate them.
- While I understand where Adam is coming from, the manner in which the gpt is laid out makes my life harder (my goal is for it to be pushed into LINUX as an alternative page table). The GPT is spread over too many directories. I realise this is to accomodate different types of GPT - even so, I feel it is a bit over the top. Scons plays a significant role and is very progressive. I have made a couple of small changes to files to build it due to the power of scons.
- Dump everything not necessary off paulaner.
15/06/05
- I have built the system with my makefiles.
- I will start my 64 bit version for LINUX tomorrow.
14/06/05
- Started building a proper Makefile to replace the scons build.
- Compiled the slab allocator and bitmap for INITIAL user space version.
- Flushed scons stuff out of allocator and bitmap.c. The way the program is set up and built with scons does not suit me.
- Read about ccache
- I have some sort of ccache problem when trying to build the 32 bit version with scons on quasar - but I don't care at the moment. FIX LATER
02/01/05
- Reading all about scons. Might play with it for a little while.
- Yuck. I am drawing diagrams, getting a feel for where things are and will go back to makefiles. No immediate interest in scons.
* (PeterChubb): Yuck is my feeling exactly about scons and arch.
01/01/05
- Building GPT on quasar with my own makefiles.
- Preliminary investigation of GPT - remind myself how it works.
18/05/05
- Finished the patch set for 2.6.12-rc4. I have written most my comments for each patch in a file. I am not going to post them to CVS until I have run some tests.
- The prelimnary patch set is in /usr/src/pauld/preliminary/patches/ on tartufi.
18/05/05
- Will check in patch set with comments for 2.6.12-rc4 late tonight.
12/05/05
- Will rework the patches again:
- The story.
1. Introduce general page table interface not including iterators. 2. Introduce all new general files required. 3. Rearrange all general code (pte_alloc_map etc). NOT CALLING interface yet. 4. Introduce iterators. 5. Call iterators. 6. Introduce page table interface specific to ia64. 7. Introduce files especially for ia64. 8. Rearrange files for ia64 fault.c init.c etc
12/05/05
- I am building the patch series along side my x86. None of the abstractions are incorrect as such - some are only relevant for ia64. I am separating out generic mlpt-interface and architecture specific interface and hooking it in.
11/05/05
- Done the patch set for rearrangement on ia64. I tried to then see how seemlessly I could bring the rearrangement across to i386. I am going to rework the patch series side by side - it helps to make the abstraction as architecture independent as possible. Its not too bad but should be far more elegant.
- The broken up series of patches will do both ia64 and i386 - where you could choose one or the other or both of the architectures.
- I do not expect to add in the abstractions until Friday.
5/05/05
- Making some changes to tartufi
28/04/05
Reviewed Peters list of changes to make to my patch. DONE
- Started to look at the assembler abstractions
- I am thinking carefully about how to break my patch up.
- I have decided to make a serious attempt at abstracting the assembler before fixing up the patch
27/04/05
- Looking at the syscall remap_file_page(), nonlinear vma etc.
- Playing with NUMA machine for the first time.
25/04/05
- Fixed my bug. I made a mistake on the abstractions of try_to_unmap_one and page_referenced_one. I had feared a subtle problem in an iterator abstraction - but they were straight forward stupid mistakes in the most simple abstractions.
22/04/05
- All functions are now abstracted in situ, in what I believe is the most elegant and efficient manner. The patch boots on both ia64 and 1386. Refer to patch pti.patch version 1.38.
- After cleaning up what I have done I will check a few remaining abstractions for correctness.
- I will then shift the functions that require shifting - pte_alloc_kernel and so on. I have done all the abstractions - a number of functions will now simply need to be moved behind the interface.
- I will also fully document the new interface on the page table interface page.
- The interface has three iterators: a read, build and read-build iterator. I believe this is the most elegant solution.
- LMbench shows very little effect on fork on either i386 or ia64 LMbench. The three iterators all operate in the same way and I expect no deterioration in performance resulting from the 10 functions that use the three iterators. There is however a degradation in performance for mmap latency of around 5% according to LMbench. I am going to look into this further - but there does not appear to be a simple solution to this. I am hoping that we will be able to live with this.
- Efforts now turn to testing.
21/04/05
- Got to the bottom of how to make remap_pfn_range fit under build iterator easily without requiring its own iterator.
- I have decided that a single iterator - while entirely doable - is not the natural abstraction.
- If I have a single iterator, I will have to pass a paramater telling it to read or write. If this is the case then I might as well have separate iterators. In my opinion it is the more elegant to have a read and build iterator.
- Alternatively, I could could have essentially just a build iterator and assume that it is a very rare occurrence that a directory will be missing - and build the missing bits that I read - (and then read them as zero) UGLY. NO WAY.
- I will revert to a single read, a single build and a dual read-build iterator. This will require a a patch back track of a couple of versions. Not a problem.
15/04/05 - 20/04/05
- I will revisit trying to combine the read and build iterator over the next couple of days.
- I have a complete patch except for remap_pfn_range. It is wrecking my symmetry. I am going to have to provide an iterator just for this rogue function.
I have a read and a build iterator that covers:
map_vm_area()
zeromap_page_range()
unmap_vm_area()
change_protection()
sync_page_range()
unuse_vma()
unmap_page_range()
try_to_unmap_cluster()
verify_pages()
- I have a dual iterator for copy_page_range().
14/04/05
- abstracted zeromap getvmarea and remap_pfn range with iterator
- I have a number of untested abstractions
- I am investigating something called VMregress to see if I can get it to run the untested abstractons.
12/04/05
Worked on PageTableInterface page
12/04/05
- anonymous_untouched_pages() FIXED
- get_user_pages() fixed.
11/04/05
- I want to benchmark the new unmap_page_range.
- Then knock off change_protection and zeromap_page_range
- Then benchmark these guys.
10/04/05
- FINALLY - I have produced an iterator that does very little to harm performance. I have dispensed with the iterator design pattern - I am passing the function that that operates on the ptes into the iterator instead.
- copy_page_range produces negligable degradation in performance according to LMbench and around 3% for the mmap latency.
- We will see how this pans out for other functions that use the iterator, but looking at the next cab off the rank, unmap_page_range - it looks straight forward.
- unmap_page_range FINISHED -page table independent
- follow_page FINISHED - page table independent
- try_to_unmap_one FINISHED - page table independent
- page_referenced_one FINISHED - page table independent
09/04/05
- The problem is the iterator design pattern is too inefficient at its core. I am investigating the use of passing functions by pointers to do the job.
- I have changed my mind. After some further tests and benchmarking I don't think saving state vrs working out a pte from the pgd and the vadrr will make a difference. Medium grain locking MUST be the answer. Medium grain locking vrs overlocking or underlocking must be the answer to the performance loss. If medium grain locking cannot improve the performance then it MAY be this is the best that can be done. I stress MAY I am not done yet. I don't have any experience in these matters do draw on - so I am trying everything I can think of.
08/04/05
- I have put a new iterator into the 2.6.12 kernel for copy_page_range(). I have benchmarked it and it gives similar results to the previous iterator. The difference between this iterator and the last is that the latest iterator does a depth first search across gaps in the table.
- I will now be looking to pick up the pace or the iterator in two ways. I still have what I call the return to root problem. I can sort this by "saving state". I believe this will give back the lost performance. The second thing I will do is inline everything.
- There is a possibliliy that some performance can be regained via medium grained locking. I have only experimented with coarse and fine grained locking. I strongly suspect that it is not a locking issue however. At the very least, with whatever tests LMbench is running I doubt locking is the problem. There was little discernable difference between coarse and fine grained locking.
- The next iteration of iterator will not only jump gaps efficiently but it won't return to root. DONE - it gave me nothing.
- The ability to jump gaps efficiently may come into play with other functions. The iterator is intended to be multi purpose. When copy page range gets a page range - it seems to be called with contigous pages (which is what I always suspected). Any gaps should be due to bad pgd, puds pmds and ptes. Thats why the depth first search didn't buy me any speed. It may do later - so I think it should stay.
07/04/05
- Put iterator in kernel. I will not be coming in today. The real iterator is finished I now have to incorporate it with copy_page_range I will then benchmark it. May not finish that today as I have to do some other things.
- I have a number of ideas for different iterators - but I hope this will give me the performance I am looking for.
05/04/05
- Working on iterator that I BELIEVE will perform acceptably.
- I think this iterator will solve the mmap latency problem with a new build_page_table() function that saves it state. It should provide greatly enhanced efficiency when adding contiguous pages. The iterator should now be efficient as well. I am looking forward to seeing the results in the next day or two.
01/04/05
- Put patches for 3 level page table in CVS
- Can't use tartufi or boot a kernel at the moment - I stuffed up changing elilo.conf file and then I couldn't reboot it from the card. In my frustration at trying to get a kernel over the network via the management card - I reset the card so I stuffed that too. I am using baci - but I don't have the priviliges so I can't copy a kernel image to /boot.
- I am updating my MLPT model to 4 levels using the PUD. I am going to build the iterator first anyway - and I will do that with my model anyway.
- The pud shouldn't hold me up more than a few days, it is irritating more than anything else. Not being able to use tartufi is far more inconvenient.
31/03/05
- Put patches for 3 level page table in CVS
- Examining the pud_t. Appears to fold straight back onto the pgd - so we still have a three level page table.
- Start rolling my stuff forward to 2.6.12.
- Make the page table interface page my own now.
17/03/05
- Working on copy page range. The use of an iterator and my new insertion function should sort this one out beautifully.
- I am now going to fix a problem in userspace then transfer it immediately to kernel space and ensure it works in the kernel immediately.
- My iterator needs to be able to cope with bad pgds, bad pmds etc...
16/03/05
- Coding three level page table with new interface.
- I can now create and delete page tables behind clean interface. for user processes. I will add the kernel to this list soon.
- But first. I am writing test code to generate test data so I can target specific regions in the address space. Although there really should be any problems because the thing is totally code pinched. I simply have to do it though.
- I also need to build the quicklists in now as well.
16/03/05
- Coding three level page table with new interface.
- Dealing with clear page tables.
16/03/05
- Coding new three level page table with new interface.
15/03/05
- Run my laptop to a repair centre in Prospect. Will be at work as soon as this is done.
- Come up with a time line for this session.
- Decided not to build a temporary page table. I will rebuild the MLPT with a new interface.
15/03/05
- Find out about Scon and understand how Adam wants me to store my stuff along side his in the kenge tree.
- Look at Adams new GPT stuff in the kenge tree.
10/02/05
Spell checking/grammar & reference report
- Add a title page and an appendix. Finalise conclusion.
- Convert to pdf and send to Darren Williams, Adam Wiggins and Peter Chubb.
6-9/02/05
- Working on report, poster and presentation.
- Expect to complete final report by tonight. 9/02/05
- Planning to send report late Wednesday night to Darren Williams for advice prior to on sending it to Adam Wiggins and Peter Chubb on Friday.
5/02/05
- Working on draft
- Working on poster
- finalise gpt test code
4/0205
- Now have a rought draft for report
- Working on test code for gpt
- fixing unmap cluster - writing iterator. Created function get_cluster_range() and put it in the dependent interface, where contents is just abstracted from unmap_cluster.
- In the process of fixing up the last C problem in the clean interface I realise I have made one slip up in patching as a consequence of now working with two patches cleaninterface.patch and gpt.patch.
- Think about poster
- FINISHED ADDING THE ITERATOR into try_to_unmap_cluster. I used Design Patterns by Grady Booch to get the job done. It really is rather nice.
3/02/05
- Working on report
- Whipped up some code to hammer Adams gpt with. Maybe do some testing later - but should be doing report.
2/02/05
- Finished get_vm_area(). I currently have a gpt attached to every process including init_mm. I call both the original get_vm_area() and now gpt_get_vm_area(), so that both page tables are built in tandem (where vmalloc is concerned).
- I will now start my report in earnest. When that is finished I will work on dumping the contents of the gpt into a file from the kernel.
1/02/05
- Found another range of addresses that Adams gpt didn't like. FIXED.
28/01/05
- Dropping the gpt into the kernel today.
- Make notes so I can make the subtle but necessary changes to be able to drop the
- gpt in and out of the kernel as seamlessly as possible for as long as possible.
- I want a dual development environment for as long as possible.
27/01/05
- Separate out patch ie. put gpt and generic interface into two patches.
- Now have an clean interface patch and a separate gpt patch. The interface patch must be applied before the gpt patch
26/01/05
- I am finding it difficult to press on with the rest of the project knowing that the gpt has so many holes in it. The more I look at the gpt the more problems I find.
- I am going to do some debuging today.
- The logic in lengthGuardMatch is wrong. anyway.
List of quick notes for myself.
- map_vm_area() required for boot. See output.
- unmap_vm_area() NOT required for boot.
- clear_page_tables NOT required for boot. (VERIFY)
- change_protection() required to get to a boot prompt. Do get printk output though Running 0dns-down to make sure resolv.conf is ok... is where it stops.
- filemap_sync() NOT required for boot.
- unuse_process() NOT required for boot.
- copy_page_range() required. Do get following prink output: INIT: no more processes left in this runlevel
- umap_page_range() required. Booted but couldn't log in.
- follow_page(): having problems shutting down without it. Otherwise not needed.
25/01/05
- Wrote up changes for Adam Wiggins.
- Building a src tree with all my changes into it. This src tree contains Adams partially debugged work, my work and infrastrucure to enable me to build the crux of my stuff in user space.
21/01/05
- Push on with debugging the page table. Found yet another problem with it last night.
The look up function "out of the box" did not work. FIXED that last night.
- wrote pte_lookup for the pagetable last night as well.
20/01/05
- Apply the pte extension patch to my kernel. I am now in the process of putting the page table into
the kernel. ON HOLD I remain unconvinced of the necessity of this as yet.
- Unfortunately there are bugs in the gpt version "out of the box". I am in the process of debugging the gpt that I have been given so that it is at least usable. As it stands, the gpt breaks down things are mapped in certain orders. If I pushed on with the gpt as it stands the kernel will never boot.
Major bug FIXED In the version on CVS there is a bug in traverseInternal.
19/01/05
- I got Adam Wiggins page table to run on a 32 bit PC this morning. It appears to only function properly for a fixed level of 1.
- I have got the 64 bit version to run. While there appeared to be a bug in the 32 bit version - the 64 bit version looks OK. It is early days yet. The 64 bit version does not look to me like it has ever run before. I had to make changes to several data structures to get it to run. In doing so I have increased the size of a node from 16 to 24 bits.
- Neither the 32 or 64 bit version appear to run on anything other than level 1. This means that the gpt really supports path compression only. The tree doesn't appear to ever have really implemented level compression - it has a mechanism to adjust 'levels' which is presumably a tempory measure.
18/01/05
- Trying to get the 32 bit page table version to compile.
17/01/05
- PC broke down this morning. Working at Lucys machine.
- Got Adam Wiggins gpt code to work off. I have been working on this all day and am definitely in business.
- Putting it together piece by piece to ensure I understand it.
14/01/05
- Quick look over page table interface to remind me of the things I need to ask.
- Looking at the data structure to be implemented - running through how how level compression and path compression work on paper. ie. Make up small address space, draw a tree, work out what the guards would be and shift the bits on paper.
- Having looked at Liedtkes paper "ADDRESS SPACE SPARSITY AND FINE GRANULARITY" I now understand guards and how to traverse a gpt. This paper provides a very clear explanation of guards.
12/01/05
- Ticked fork.c off as being interface independent. While it refers to pgds and allocates pgd and frees pgds - pgalloc.h is implementation dependent - which is where all calls come from.
- Looked an handle_mm_fault(). I am adding it to the external interface and shifting all of its helper functions across into fixed_3_level.c
11/01/05
- I have rectified my patching problems - and repaired my patch.
- MAJOR PRIORITY. Cleaning up the interface has involved the shifting of many functions. In my opinion, as a matter of safety 2.6.10 should be benchmarked against the 'new interface' prior to implementing the new page table. Whilest I have been as careful as possible in the reshuffle - I suffer from paranoia.
- Whilest the interface is basically finished, I am still examining it carefully. The full scope of the job at hand is becoming apparent.
- I intend to properly document the new interface when I have discussed it in detail with Peter Chubb - and I am 100% happy. Despite being happy with my work thus far I am simply not yet satisfied.
- I intend Lucy's page table interface page to properly reflect the interface. It requires considerable updating and clarification. This is on my TODO list.
10/01/05
- Fixing the patch I added to the repository. I have been using quilt to patch and I had an issue adding new files to the tree. I was adding the file, then going quilt add myfile.c and then going quilt refresh. Consequently the current patch expects to find the file already there. I need to do 'quilt add myfile.c', then add the file. Just another little frustration...
- I hope to resubmit a patch that will take properly later today, but I am feeling unwell so I am going home to take a break for a couple of hours. Then I will finalise things from home.
09/01/05
- Fixed void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *prev, unsigned long start, unsigned long end) to interface. Abstracted out implementation dependent functionality into function clear_pt_tlb_wrapper();
- clear_pt_tlb_wrapper() to interface.
08/01/05
- Fixed int handle_mm_fault(struct mm_struct *mm, struct vm_area_struct * vma,
- unsigned long address, int write_access) to interface. Now calls pte_alloc()
- Fixed struct page *put_kernel_page (struct page *page, unsigned long address, pgprot_t pgprot); Now calls pte_alloc_k()
- Working on: static int move_one_page(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr) Function is fine: Left it where it is.
Still need to look at fork.OK Fork.c is absolutely fine. pgd's and functions from pgalloc.h are called.
07/01/05
- I want to finalise a patch that results in a source tree with a finalised interface that is ready for the the page table implementation to be compiled in bit by bit. So I am going to pull back on the interesting stuff for today - and finalise my patch.
PATCH DESCRIPTION Will be finalised by weekend.
06/01/05
- Compiling Cristans code - in user space first and then in OS.
- Revisiting something about memory allocation of page tables that is bothering me
- I was doing some more cleaning up of the interface when I noticed a couple of lines missing in a function that was moved from memory.c to fixed_3_lvl.c. It was somehow the result of my hand patching gone wrong because it was one of the functions that has appeared since Lucy prepared the interface. I have no idea now it happened but the lines have been added back in.
05/01/05
- Finished configuring system so I can compile in or out fixed 3 level page table or variable radix page table.
- Read about variable radix page tables and got a partial implementation of the variable radix page table written by Cristan Szamdja.
- Going through Lucys interface function by function and writing an explanation of each function in the interface. The purpose of this is to gain a full understanding of each function in the interface, so I can produce a minimal set of functions in the interface to work on with regards the variable radix page table. Hopefully I can then take the minimall set and get the OS to boot. This will be my starting point.
- Members of set include: map_vm_area, unmap_vm_area, a page table initialisation function and follow_page.
04/01/05
Upon having a good look at clear_page_tables I have found that it is not OK to pass struct mm_struct
instead of struct mmu_gather. A flag within the mmu_gather structure is set - which appears to be
used later for tlb flushing. As a consequence - I think struct mmu_gather should be added to Lucy's
list of parameters that can be passed across the interface. I have discussed it with Darren and he
confirms that this is OK.Upon closer examination of passing pgprot_t prot across the interface, I have decided that this
should also be recognised as another parameter which can be passed across the interface. While it is
possible to wrap it up in a pte - I think that it is definitely not the right thing to do. Darren
has confirmed that this too is OK.As it stands, the files have an implementation independent interface, and functions are now grouped
in appropriate files.OUTSTANDING interface issues as I see them...
- Still to change pgd directory pointer from pgd to generic gpt pointer.
clear_page_tables() is passed the number of pgds to 'clear', and I feel that this needs to be
changed. I have spoken to Darren about it - but he felt that my plan to pass a start and end address
was no good owing to SuperPages. We are still to decide what to do about this.- I am now going to add in a menu switch to choose between compiling in the RVPT or the 3 level page table.
30/12/04
To conform to Lucys interface I am planning to change:
From: void clear_page_tables(struct mmu_gather *tlb, unsigned long first, int nr)
To: void clear_page_tables(struct mm_struct *mm, unsigned long first, unsigned last)
From: int map_vm_area(struct vm_struct *area, pgprot_t prot, struct page ***pages)
To: int map_vm_area(struct vm_struct *area, pte_t pte, struct page ***pages)
From: static int map_area_pmd(pmd_t *pmd, unsigned long address, unsigned long size, pgprot_t prot, struct page ***pages)
To: static int map_area_pmd(pmd_t *pmd, unsigned long address, unsigned long size, pte_t pte, struct page ***pages)
From: change_protection(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgprot_t newprot)
To: change_protection(struct vm_area_struct *vma, unsigned long start, unsigned long end, pte_t pte)
From: static void unmap_page_range(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long address, unsigned long end, struct zap_details *details) To: static void unmap_page_range(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned long end, struct zap_details *details) Still to look at struct zap_details.
more changes will follow ...
The following already conform to the interface:
void unmap_vm_area(struct vm_struct *area)
int copy_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *vma)
struct page *follow_page(struct mm_struct *mm, unsigned long address, int write) Although int write doesn't conform with the interface it must stay, but it is implementation independent.
static inline int page_table_present(struct mm_struct *mm, unsigned long addr)
static inline int untouched_anonymous_page(struct mm_struct* mm, struct vm_area_struct *vma, unsigned long address)
The above are a quick scan through of part of the interface only. More to follow...
28/12/04
I am expecting to rework pte_lookup(struct mm_struct *mm, unsigned long address) which was added by Lucy Chubb for the new interface. Lucy intended for it to be rejigged as she commented it thus. I will be replacing the page table dependent code with a call to pte_lookup in follow_page.
- I will knock a couple more functions off Lucy's TODO list and then I will post another patch in CVS repository.
- I still have some thinking to do, but the ball is rolling.
27/12/04
For the last two days I have been trawling through the page table interface with a fine tooth comb. I have built myself a small model to enhance my understanding. I have also been closely examining Lucy's work on the PTI interface. Lucy has added several functions to the new interface already but they are yet to be used. I may incorporate them over the next few days - if I don't get side tracked.
- Need to think about dealing with hugetlb file system.
23/12/04
- Patch added to CVS repository as testpatch.patch. This incorporates Lucy's page table interface work into the latest kernel.
- Rough explanation for how patch was generated in testpatch.README.
- I will rename the patch and put it in the right place in the repository at some time over Christmas. I want to get on with the job as opposed to fiddling with CVS for the moment.
- THE PATCH transfers Lucys work as closely as possible to the latest kernel. There were a handful of changes between kernels causing the need for manual patching. A couple of extra functions, altered functions, depracated functions, shifted function implementations etc - but nothing of any real consequence.
- The next iteration of the patch will kill warnings, deal with some prototyping issues - two functions still to be shifted.
22/12/04
- Finished my patch porting lucys work on the page table interface to the latest kernel release.
- Obviously it compiles and reboots fine.
- Will do the readme tomorrow (probably) and put the patch and associated readme in the CVS repository.
- The readme will explain exactly what I did to port the patch from the six month old kernel to the latest kernel.
- The differences are in fact extremely slight - it just took me a little time to figure out how to approach manual patching when the patch doesn't take (after coming to terms with quilt). In retrospect it was extremely simple, but I am far better for the experience and expect to be able to carry out a similar task far more quickly in the future.
- NB: When not at work I tend to work at night and into the early hours of the morning ie. I am only at my PC late afternoon and night.
17/12/04
Paul will be away from 20/12/04 until 03/01/05. I will be updating the wiki from time to time over the next couple of weeks.
- Still working on producing my patch
- Definitely finished ironing out all my quilting problems
- Expect to have my first patch in the page table interface CVS repository some time next week
16/12/04
- chasing changes between kernel Lucy's kernel version and the current kernel version to get it to compile.
- Gone out for an hour
- Skipped BBQ to finish my patch
15/12/04
- Still getting the patches to take to the latest vanilla kernel
- Created a CVS repository for the page table interface.
14/12/04
- Dealing with Lucy's patches
- Extracting a suitable patch from a suitable changeset and applying it to the latest src tree
- Compiling with new patches is a work in progress and will continue into tomorrow.
Now I really am getting to know BitKeeper
- Kernel including Lucys 1.1784 patch compiles and boots.
- Kernel with Peters 1.1785 patch now compiles and boots
13/12/04
- Progress Report and meeting
- Looking at some patches
- Reading ia64Linux
10/12/04
- Reading ia64Linux
- Listen to talk by Darren and Ian on VHPT
9/12/04
I have decided to take both a top down and bottom up approach to the project. On the one hand I am reading all about the linux VM system which is extremely involved. I have decided to combine this with some experimental code to allow me to get a concrete understanding of the page table and to meet the interface. I want to develop a genuine feel for the page table and its interface and I think knocking up some experimental code is the way to do it.
- Get to know the page table interface by experimenting with it.
- Play with the beast!
8/12/04
- Reading ia64Linux chapter on VM.
- Checking out bits and pieces of the source in conjuction with this.
Going out for an hour at 3:45PM.
7/12/04
Source familiarisation: Examined some key data structures In Progress
Start trawling through ia64Linux chapter on VM Not DONE - sidetracked
- Looked at paper on Page Tables by Mathew Chapman instead.
- Gain much needed proficiency in traversing the C source tree effectively.
Now using cscope DONE
Apply Lucys patches (from Peter). Not DONE - side tracked
- Think about the lvhpt.
- Where it fits in.
How and why does it effects the the disentagling of the interface. In Progress
Overview of Day
Trying to move away from the mechanics of working in the new environment to thinking about the project itself. Far more interesting.
6/12/04
Compile an unpatched kernel and boot to tartufi such that I am able to ssh into tartufi. Done
Have a working config by the end of the day. Done
Overview of Day
Kernel is up and running. Time to start the actual project...
3/12/04
- Feel my way around the src tree for a while (very little time spent on this task)
- Come up with a stripped down config for building kernel. (Took most of the day)
2/12/04
- Install Debian
- Personalise workstation
- Start reading article on radix tables
Overview of Day
Debian successfully installed - unstable version. One major outstanding issue yet to be resoved. NIS is not working properly yet.
1/12/04
The network for my workstation is unavailable at the moment - so I am in drum learning to use the wiki and reading various documentation relevant to the project.
- Learn all about the wikiwiki
- Utilise the wikiwiki in the spirit intended by the powers that be
- Work on my understanding of the project
- Begin detailed development of plan of attack - to be posted on wiki within a few days.
- To do this - read about the Variable Radix Page Table
- Closely examine Lucy Chubbs written requirements for a clean interface to the page table
- more details on this in tomorrows todo list.
Overview of Day
My cse account was unavailable after around 3:30 while I waited for maintenace to be done.
30/11/04
Introduction to workplace Done
Account setup Done
Download and compile kernel Done
Patch kernel with lvhpt Done
Examine ski simulator Done
Overview of Day
All in all a successful day. I now have reference point for the project ie. I can download, patch the kernel - and compile. I have yet to successfully boot the new kernel image to tartufi. This will go onto the top of my todo list. I still require much more time coming to terms with the environment. My C coding is much stronger than knowledge of the linux environment, however I aim to be comfortable (to my satisfaction) with my development environment by Monday.
