Cavan, This is the feedback that we have reviewed from ARACY in regards to the DataFerrett evaluation that we undertook. We should discuss when we get a chance. Also, am updating the TWIKI. Some of it is already done. Regards Wayne Richards Australian Bureau of Statistics 02 6252 7797

---

Forwarded by Lorraine Cornehls/Staff/ABS on 14/03/2007 10:03 AM ----- |--------------------------------->

David Lawrence

D.Lawrence@curtin.edu

.au

08/03/2007 06:36 PM

|---------------------------------> >----------------------------------------------------------------------------------------------------------------------|

To

lorraine.cornehls@abs.gov.au

cc

trevor.draper@abs.gov.au, Mark Bingley <mark.bingley@abs.gov.au>

Subject

Re: Undertaking for DataFerrett

Protective Mark

>----------------------------------------------------------------------------------------------------------------------|

Dear Lorraine,

Thanks for your message.

I can confirm that DataFerrett has been deleted from my computer. I did not store any data during the testing.

I showed the DataFerrett application to Fiona and Steve (Zubrick). We all felt that the addition of this type of tool is an important part of development of the NDN. Extending the NDN beyond the metadata repository to include tools and facilities that would allow custodians to make confidentialised microdata files available, and provide researchers and interested users with the tools to be able to query those files will very much enhance the value of the NDN toolset, and is an important step in realising the NDN vision.

Keeping in mind that it is a beta test software, we did have some concerns about the usability of the interface. We feel that it should be possible to simplify the design, and make it more user friendly. For instance, there seems to be more steps involved in selecting the variables you want to tabulate that you would expect. The interface could benefit from using a design with more similarities to existing software. Perhaps a usability testing phase would be useful.

The graphing facility seems hit and miss. Obviously you need to select an appropriate area in the spreadsheet that suits the requirements of what DataFerrett will graph and what it won't. More feedback for the user would be helpful. Our experience was that when you select some data to graph there is a long response lag before the graph will appear. If you don't get your selection right, the graph just doesn't appear. Which means you wait wondering if you have made an error in selection or the graph is just slow to appear. I'm sure with appropriate experience and knowledge of the tool this would be less of an issue. More feedback from the DataFerrett on what it is doing would help the user. Perhaps a roles-based graphing wizard of the type employed in major statistical packages (or even Microsoft Excel) would be a more familiar type of interface design, at least for users who have a background in that sort of thing.

We realise it is a beta software, but the slowness of response was a concern, as I imagine it is not being heavily loaded at this stage. I hope this reflects some configuration aspect of the test machine rather than some underlying deficiency in the software code.

Issues that we couldn't evaluate but we feel are important are:

1) Security - we imagine that before custodians are willing to expose microdata files via the DataFerrett it would be necessary to have some level of assurance or testing that the facility is secure and resistant to attempts to hack into it. I don't know exactly how this could be achieved, but I imagine an important first step would be seeing if the ABS is willing to expose any of its data sets via the facility, and this would be an important confidence booster to help bring other custodians on board.

2) Confidentiality / Privacy protection - it wasn't obvious that the DataFerrett has any inbuilt facility to suppress the tabulation of small cells that might pose a confidentiality risk. I know that the test data was not at the unit level, but if unit data were exposed it would be important to have a facility to suppress small cells. As different organisations apply differing rules on disclosure control, it may be necessary to develop a national standard in this regard, or to allow the custodian some flexibility in how rigorously this would be applied.

3) Sampling error - if the microdata being tabulated is from a sample survey rather than from a census, is there a facility to calculate any measures of accuracy of weighted figures, such as standard errors or RSEs or confidence intervals? I get the impression the answer is no. We do feel this would be an important aspect of the facility. Most of our own data holdings are in the form of surveys rather than censuses. Estimates of precision could be particularly important given that many non-ABS sources are likely to use smaller sample sizes and have less statistical power than standard ABS collections. Preventing users from making poor decisions based on inappropriate conclusions from weighted sample data that may have high variability would be an important objective.

Areas for future development

1) We thought that some custodians may like to have some facility to control what tables users can request against their data sets. For instance, by restricting the number of cross-classifications that can be run at a time, or by preventing certain fine-grained variables from being crossed against each other.

2) The facility for generating tables and graphs, particularly the graphs, may be enhanced with the ability to perform some basic analyses on the data, such as correlations or fitting a regression line.

3) A downloadable or printable manual, perhaps in PDF form, would be a welcome addition to the HTML based help pages.

I hope this information is of some help.

Best wishes,

David.

lorraine.cornehls@abs.gov.au wrote: Hi David,

Thank you for undertaking the evaluation of DataFerrett. At this stage I am not aware of any plans to release a new version of DataFerrett.

Could you please confirm that you have complied with the conditions stated in the Personal Confidentiality Undertaking, in particular, that you have returned all information to the ABS and removed DataFerrett from you local hard drive?

In addition, would also you be able to provide a brief wrap up of the results of your evaluation?

Regards,

Lorraine Cornehls (02) 6252 6079

David Lawrence

D.Lawrence@curtin.edu

.au To

lorraine.cornehls@abs.gov.au

19/02/2007 07:18 PM cc

Subject Re: Undertaking for DataFerrett Protective Mark

Dear Lorraine,

Thanks for your message. I would say that unless there are plans to release any new versions of the dataferrett with additional functionality that it wouldn't be necessary to extend the date for the confidentiality undertaking.

Best wishes,

David.

lorraine.cornehls@abs.gov.au wrote:

Hi David,

In November 06 we arranged a Personal Confidentiality Undertaking for you in relation to the evaluation of DataFerrett using ABS data. The undertaking you signed will expire on 28 February 2007. Could you please let me know if you need to continue with your work beyond this date?

Regards,

Lorraine Cornehls (02) 6252 6079

------------------------------------------------------------------------------------------------

Free publications and statistics available on www.abs.gov.au

-- David Lawrence

Centre for Developmental Health Curtin University of Technology Telethon Institute for Child Health Research PO Box 855 West Perth WA 6872

Ph

+61 8 9360 5811

Fax

+61 8 9489 7700

Email

D.Lawrence@curtin.edu.au

-- David Lawrence

Centre for Developmental Health Curtin University of Technology Telethon Institute for Child Health Research PO Box 855 West Perth WA 6872

Ph: +61 8 9360 5811 Fax: +61 8 9489 7700 Email: D.Lawrence@curtin.edu.au

---

Free publications and statistics available on www.abs.gov.au

-- CavanCapps - 07 May 2007