The agony and the ecstacy: Development of complex MATRIX programs.

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

The agony and the ecstacy: Development of complex MATRIX programs.

David Marso
Administrator
This is not so much a question as much as a general comment and I am interested in other people's experiences.
Context:  
SPSS has had an embedded MATRIX language for over 20 years.
For those with the necessary mathematical background and special needs it is a wonderful tool.
However:  
The documentation SUCKS!
Error handling is NON EXISTENT so debugging anything more than trivial programs is obscenely difficult.
There have been absolutely NO enhancements in the past 20 years.
There is no concept of modularity (no support for subroutines/local variables).
I would like to see IBM/SPSS development open up the can of worms called MATRIX and reexamine its potential for developers to create specialized solutions.
JoNoH... Please don't tell me to use Python.  That would be passing the buck.
Don't tell me to use R " "...
Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind the game.
----


Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Jon K Peck
Well, MATRIX could use a lot of enhancements, but I will tell you that using numpy via Python programmability is a thing of beauty, and it has all the error handling, modularity, expressive power, and functionality that any user of such a procedure could want.  Here's the top level of the reference manual:
http://docs.scipy.org/doc/numpy/reference/index.html#reference

No doubt, the current MATRIX procedure, while powerful, is lacking in a number of ways, but modernizing it would be a major effort - pretty much a rewrite from scratch.  And I am sure that a rewrite would look quite different from the current syntax, so it would have to be a different procedure.  When there are free, readily available, and efficient add-ins such as numpy, is this really where IBM/SPSS should be spending its scare development resources?  It's not a matter of passing the buck: it's an issue of making the best use of SPSS resources.

My unofficial two cents worth.

Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: 720-342-5621




From:        David Marso <[hidden email]>
To:        [hidden email],
Date:        12/10/2012 07:40 AM
Subject:        [SPSSX-L] The agony and the ecstacy:  Development of complex              MATRIX              programs.
Sent by:        "SPSSX(r) Discussion" <[hidden email]>




This is not so much a question as much as a general comment and I am
interested in other people's experiences.
Context:
SPSS has had an embedded MATRIX language for over 20 years.
For those with the necessary mathematical background and special needs it is
a wonderful tool.
However:
The documentation SUCKS!
Error handling is NON EXISTENT so debugging anything more than trivial
programs is obscenely difficult.
There have been absolutely NO enhancements in the past 20 years.
There is no concept of modularity (no support for subroutines/local
variables).
I would like to see IBM/SPSS development open up the can of worms called
MATRIX and reexamine its potential for developers to create specialized
solutions.
JoNoH... Please don't tell me to use Python.  That would be passing the
buck.
Don't tell me to use R " "...
Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind
the game.
----






-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD


Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

David Marso
Administrator
Translation:
IBS/SPSS could care less about advanced users needs or keeping up with the Joneses in that respect and everyone who wishes a modern programming environment for designing/implementing bleeding edge statistical analytical techniques should learn Python and require the consumers of such developments to install and maintain python.  Since only about 1 percent of SPSS users could care less about such capabilities such considerations shall be deferred for the foreseeable future ;-)
---
Maybe a middle ground?  Place certain useful functionality such as various useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc) into a callable library and open up MATRIX to be able to call these functions.  That would NOT require a complete rewrite.
 
Jon K Peck wrote
Well, MATRIX could use a lot of enhancements, but I will tell you that
using numpy via Python programmability is a thing of beauty, and it has
all the error handling, modularity, expressive power, and functionality
that any user of such a procedure could want.  Here's the top level of the
reference manual:
http://docs.scipy.org/doc/numpy/reference/index.html#reference

No doubt, the current MATRIX procedure, while powerful, is lacking in a
number of ways, but modernizing it would be a major effort - pretty much a
rewrite from scratch.  And I am sure that a rewrite would look quite
different from the current syntax, so it would have to be a different
procedure.  When there are free, readily available, and efficient add-ins
such as numpy, is this really where IBM/SPSS should be spending its scare
development resources?  It's not a matter of passing the buck: it's an
issue of making the best use of SPSS resources.

My unofficial two cents worth.

Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: 720-342-5621




From:   David Marso <[hidden email]>
To:     [hidden email],
Date:   12/10/2012 07:40 AM
Subject:        [SPSSX-L] The agony and the ecstacy:  Development of
complex              MATRIX              programs.
Sent by:        "SPSSX(r) Discussion" <[hidden email]>



This is not so much a question as much as a general comment and I am
interested in other people's experiences.
Context:
SPSS has had an embedded MATRIX language for over 20 years.
For those with the necessary mathematical background and special needs it
is
a wonderful tool.
However:
The documentation SUCKS!
Error handling is NON EXISTENT so debugging anything more than trivial
programs is obscenely difficult.
There have been absolutely NO enhancements in the past 20 years.
There is no concept of modularity (no support for subroutines/local
variables).
I would like to see IBM/SPSS development open up the can of worms called
MATRIX and reexamine its potential for developers to create specialized
solutions.
JoNoH... Please don't tell me to use Python.  That would be passing the
buck.
Don't tell me to use R " "...
Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind
the game.
----






-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to
email me.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Jon K Peck
I disagree completely with Marso's so-called "translation".


Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: 720-342-5621




From:        David Marso <[hidden email]>
To:        [hidden email],
Date:        12/10/2012 09:03 AM
Subject:        Re: [SPSSX-L] The agony and the ecstasy:  Development of complex              MATRIX              programs.
Sent by:        "SPSSX(r) Discussion" <[hidden email]>




Translation:
IBS/SPSS could care less about advanced users needs or keeping up with the
Joneses in that respect and everyone who wishes a modern programming
environment for designing/implementing bleeding edge statistical analytical
techniques should learn Python and require the consumers of such
developments to install and maintain python.  Since only about 1 percent of
SPSS users could care less about such capabilities such considerations shall
be deferred for the foreseeable future ;-)
---
Maybe a middle ground?  Place certain useful functionality such as various
useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc) into
a callable library and open up MATRIX to be able to call these functions.
That would NOT require a complete rewrite.


Jon K Peck wrote
> Well, MATRIX could use a lot of enhancements, but I will tell you that
> using numpy via Python programmability is a thing of beauty, and it has
> all the error handling, modularity, expressive power, and functionality
> that any user of such a procedure could want.  Here's the top level of the
> reference manual:
>
http://docs.scipy.org/doc/numpy/reference/index.html#reference
>
> No doubt, the current MATRIX procedure, while powerful, is lacking in a
> number of ways, but modernizing it would be a major effort - pretty much a
> rewrite from scratch.  And I am sure that a rewrite would look quite
> different from the current syntax, so it would have to be a different
> procedure.  When there are free, readily available, and efficient add-ins
> such as numpy, is this really where IBM/SPSS should be spending its scare
> development resources?  It's not a matter of passing the buck: it's an
> issue of making the best use of SPSS resources.
>
> My unofficial two cents worth.
>
> Jon Peck (no "h") aka Kim
> Senior Software Engineer, IBM

> peck@.ibm

> new phone: 720-342-5621
>
>
>
>
> From:   David Marso &lt;

> david.marso@

> &gt;
> To:

> SPSSX-L@.uga

> ,
> Date:   12/10/2012 07:40 AM
> Subject:        [SPSSX-L] The agony and the ecstacy:  Development of
> complex              MATRIX              programs.
> Sent by:        "SPSSX(r) Discussion" &lt;

> SPSSX-L@.uga

> &gt;
>
>
>
> This is not so much a question as much as a general comment and I am
> interested in other people's experiences.
> Context:
> SPSS has had an embedded MATRIX language for over 20 years.
> For those with the necessary mathematical background and special needs it
> is
> a wonderful tool.
> However:
> The documentation SUCKS!
> Error handling is NON EXISTENT so debugging anything more than trivial
> programs is obscenely difficult.
> There have been absolutely NO enhancements in the past 20 years.
> There is no concept of modularity (no support for subroutines/local
> variables).
> I would like to see IBM/SPSS development open up the can of worms called
> MATRIX and reexamine its potential for developers to create specialized
> solutions.
> JoNoH... Please don't tell me to use Python.  That would be passing the
> buck.
> Don't tell me to use R " "...
> Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind
> the game.
> ----
>
>
>
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to
> email me.
> --
> View this message in context:
>
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
>
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD


Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstacy: Development of complex MATRIX programs.

Kirill Orlov
In reply to this post by David Marso
David you are smart. I support you fully with regard to MATRIX. I take your side and not Jon's, this time.
More generally, SPSS Inc seem to have an annoying habit to drift new, discordly bedded features over old functionality. They shun the delicate and noble work to reconsider and enhance older procedures. The package has thus become a bloated layered snowdrift. Python and other Extensions give great power - who argues? - but using these totally alien languages is esthetically sickening. Now that SPSS had brought them in as they wanted it - they should feel pleased. And stop and look back. Today time is coming to turn back to native SPSS devices such as MATRIX and macros and to upgrade them cleverly.


10.12.2012 18:39, David Marso пишет:
This is not so much a question as much as a general comment and I am
interested in other people's experiences.
Context:
SPSS has had an embedded MATRIX language for over 20 years.
For those with the necessary mathematical background and special needs it is
a wonderful tool.
However:
The documentation SUCKS!
Error handling is NON EXISTENT so debugging anything more than trivial
programs is obscenely difficult.
There have been absolutely NO enhancements in the past 20 years.
There is no concept of modularity (no support for subroutines/local
variables).
I would like to see IBM/SPSS development open up the can of worms called
MATRIX and reexamine its potential for developers to create specialized
solutions.
JoNoH... Please don't tell me to use Python.  That would be passing the
buck.
Don't tell me to use R " "...
Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind
the game.
----






-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD



Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

bdates
In reply to this post by David Marso
David,

It doesn't seem like many people are responding, but I want to put in my
two cent's worth and echo your frustration.  I work almost exclusively
in matrix, and the functionality has fallen increasingly behind with
each new version release.  While admittedly, those of us who make
significant use of the matrix functionality are surely in the minority,
I think we deserve at least some enhancements to the process, e.g.,
those you mention in the previous post.

Brian

-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to
email me.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-
Development-of-complex-MATRIX-programs-tp5716810p5716819.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstacy: Development of complex MATRIX programs.

Bruce Weaver
Administrator
In reply to this post by Kirill Orlov
Here's some data concerning the description of SPSS as "bloated".  On my laptop:

IBM SPSS Statistics 20:  Size = 722 MB*
Stata 12:  Size = 270 MB

* Not including Python and R and whatever else might be "required" in future.  ;-)


Kirill Orlov wrote
David you are smart. I support you fully with regard to MATRIX. I take
your side and not Jon's, this time.
More generally, SPSS Inc seem to have an annoying habit to drift new,
discordly bedded features over old functionality. They shun the delicate
and noble work to reconsider and enhance older procedures. The package
has thus become a bloated layered snowdrift. Python and other Extensions
give great power - who argues? - but using these totally alien languages
is esthetically sickening. Now that SPSS had brought them in as they
wanted it - they should feel pleased. And stop and look back. Today time
is coming to turn back to native SPSS devices such as MATRIX and macros
and to upgrade them cleverly.


10.12.2012 18:39, David Marso ?????:
> This is not so much a question as much as a general comment and I am
> interested in other people's experiences.
> Context:
> SPSS has had an embedded MATRIX language for over 20 years.
> For those with the necessary mathematical background and special needs it is
> a wonderful tool.
> However:
> The documentation SUCKS!
> Error handling is NON EXISTENT so debugging anything more than trivial
> programs is obscenely difficult.
> There have been absolutely NO enhancements in the past 20 years.
> There is no concept of modularity (no support for subroutines/local
> variables).
> I would like to see IBM/SPSS development open up the can of worms called
> MATRIX and reexamine its potential for developers to create specialized
> solutions.
> JoNoH... Please don't tell me to use Python.  That would be passing the
> buck.
> Don't tell me to use R " "...
> Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind
> the game.
> ----
>
>
>
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to email me.
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

David Marso
Administrator
In reply to this post by Jon K Peck
You are perhaps too close to the vat of Python Kool Aid...
The consistent paradigm seems to be to defer to Python to do the heavy lifting and hence offload the economics of maintaining and or enhancing traditional methods of custom implementation from internal developers to python extensions... How long before SPSS itself is merely a collection of python extensions?
Is the internal design of MATRIX so intractible so as to make calling external libraries prohibited or providing for localized subroutines/local variables?
Is the internal design of the SPSS core transformation funstionality in such a disarray that coordinating a callable library of COMPUTE functions constitutes a herculean undertaking?
MACRO is deprecated!  Why is that so?  Why should it be necessary to resort to external development tools rather than have those tools available in the SPSS application itself?


Jon K Peck wrote
I disagree completely with Marso's so-called "translation".


Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: 720-342-5621




From:   David Marso <[hidden email]>
To:     [hidden email],
Date:   12/10/2012 09:03 AM
Subject:        Re: [SPSSX-L] The agony and the ecstasy:  Development of
complex              MATRIX              programs.
Sent by:        "SPSSX(r) Discussion" <[hidden email]>



Translation:
IBS/SPSS could care less about advanced users needs or keeping up with the
Joneses in that respect and everyone who wishes a modern programming
environment for designing/implementing bleeding edge statistical
analytical
techniques should learn Python and require the consumers of such
developments to install and maintain python.  Since only about 1 percent
of
SPSS users could care less about such capabilities such considerations
shall
be deferred for the foreseeable future ;-)
---
Maybe a middle ground?  Place certain useful functionality such as various
useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc)
into
a callable library and open up MATRIX to be able to call these functions.
That would NOT require a complete rewrite.


Jon K Peck wrote
> Well, MATRIX could use a lot of enhancements, but I will tell you that
> using numpy via Python programmability is a thing of beauty, and it has
> all the error handling, modularity, expressive power, and functionality
> that any user of such a procedure could want.  Here's the top level of
the
> reference manual:
> http://docs.scipy.org/doc/numpy/reference/index.html#reference
>
> No doubt, the current MATRIX procedure, while powerful, is lacking in a
> number of ways, but modernizing it would be a major effort - pretty much
a
> rewrite from scratch.  And I am sure that a rewrite would look quite
> different from the current syntax, so it would have to be a different
> procedure.  When there are free, readily available, and efficient
add-ins
> such as numpy, is this really where IBM/SPSS should be spending its
scare
> development resources?  It's not a matter of passing the buck: it's an
> issue of making the best use of SPSS resources.
>
> My unofficial two cents worth.
>
> Jon Peck (no "h") aka Kim
> Senior Software Engineer, IBM

> peck@.ibm

> new phone: 720-342-5621
>
>
>
>
> From:   David Marso <

> david.marso@

> >
> To:

> SPSSX-L@.uga

> ,
> Date:   12/10/2012 07:40 AM
> Subject:        [SPSSX-L] The agony and the ecstacy:  Development of
> complex              MATRIX              programs.
> Sent by:        "SPSSX(r) Discussion" <

> SPSSX-L@.uga

> >
>
>
>
> This is not so much a question as much as a general comment and I am
> interested in other people's experiences.
> Context:
> SPSS has had an embedded MATRIX language for over 20 years.
> For those with the necessary mathematical background and special needs
it
> is
> a wonderful tool.
> However:
> The documentation SUCKS!
> Error handling is NON EXISTENT so debugging anything more than trivial
> programs is obscenely difficult.
> There have been absolutely NO enhancements in the past 20 years.
> There is no concept of modularity (no support for subroutines/local
> variables).
> I would like to see IBM/SPSS development open up the can of worms called
> MATRIX and reexamine its potential for developers to create specialized
> solutions.
> JoNoH... Please don't tell me to use Python.  That would be passing the
> buck.
> Don't tell me to use R " "...
> Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years
behind
> the game.
> ----
>
>
>
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to
> email me.
> --
> View this message in context:
>
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html

>
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to
email me.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstacy: Development of complex MATRIX programs.

David Marso
Administrator
In reply to this post by Bruce Weaver
SPSS 11.5:
ALL MODULES!!!
168 MB
---
Also AFAIK:  SPSS installs a bunch of stuff in other places other than the SPSS folder.
I recently installed a SPSS 20 trial and it wanted something like 2G!
--
Bruce Weaver wrote
Here's some data concerning the description of SPSS as "bloated".  On my laptop:

IBM SPSS Statistics 20:  Size = 722 MB*
Stata 12:  Size = 270 MB

* Not including Python and R and whatever else might be "required" in future.  ;-)


Kirill Orlov wrote
David you are smart. I support you fully with regard to MATRIX. I take
your side and not Jon's, this time.
More generally, SPSS Inc seem to have an annoying habit to drift new,
discordly bedded features over old functionality. They shun the delicate
and noble work to reconsider and enhance older procedures. The package
has thus become a bloated layered snowdrift. Python and other Extensions
give great power - who argues? - but using these totally alien languages
is esthetically sickening. Now that SPSS had brought them in as they
wanted it - they should feel pleased. And stop and look back. Today time
is coming to turn back to native SPSS devices such as MATRIX and macros
and to upgrade them cleverly.


10.12.2012 18:39, David Marso ?????:
> This is not so much a question as much as a general comment and I am
> interested in other people's experiences.
> Context:
> SPSS has had an embedded MATRIX language for over 20 years.
> For those with the necessary mathematical background and special needs it is
> a wonderful tool.
> However:
> The documentation SUCKS!
> Error handling is NON EXISTENT so debugging anything more than trivial
> programs is obscenely difficult.
> There have been absolutely NO enhancements in the past 20 years.
> There is no concept of modularity (no support for subroutines/local
> variables).
> I would like to see IBM/SPSS development open up the can of worms called
> MATRIX and reexamine its potential for developers to create specialized
> solutions.
> JoNoH... Please don't tell me to use Python.  That would be passing the
> buck.
> Don't tell me to use R " "...
> Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind
> the game.
> ----
>
>
>
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to email me.
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Kirill Orlov
In reply to this post by David Marso
Again, I can't help agreeing with almost every David's angry lunge. Because it is all just, and so long sore.

And macros, too. The idea of a macro is basically beautiful: a language that generates code. It *can* be developed and enhanced, I believe, if development is willing and thoughtful, not lazy or urgent.

SPSS development has become - to my subjective feeling - so much negligent that they didn't allowed using GPL within macros - just because GPL doesn't support quoting with apostrophes (which macro function !quote does). Also, SPSS stopped developing Chart Builder (which yet never borrowed some really good features of Interactive Graphs dialog box); instead, they added yet a new feature, Graphboard Template. Now we have to wander between the Builder and the Template. Etc etc. One can continue pulling examples of this compoundness without unity, a characteristic of modern SPSS Statistics.

10.12.2012 20:54, David Marso пишет:
You are perhaps too close to the vat of Python Kool Aid...
The consistent paradigm seems to be to defer to Python to do the heavy
lifting and hence offload the economics of maintaining and or enhancing
traditional methods of custom implementation from internal developers to
python extensions... How long before SPSS itself is merely a collection of
python extensions?
Is the internal design of MATRIX so intractible so as to make calling
external libraries prohibited or providing for localized subroutines/local
variables?
Is the internal design of the SPSS core transformation funstionality in such
a disarray that coordinating a callable library of COMPUTE functions
constitutes a herculean undertaking?
MACRO is deprecated!  Why is that so?  Why should it be necessary to resort
to external development tools rather than have those tools available in the
SPSS application itself?



Jon K Peck wrote
I disagree completely with Marso's so-called "translation".


Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM

      
[hidden email]

      
new phone: 720-342-5621




From:   David Marso &lt;

      
david.marso@

      
&gt;
To:

      
[hidden email]

      
,
Date:   12/10/2012 09:03 AM
Subject:        Re: [SPSSX-L] The agony and the ecstasy:  Development of
complex              MATRIX              programs.
Sent by:        "SPSSX(r) Discussion" &lt;

      
[hidden email]

      
&gt;



Translation:
IBS/SPSS could care less about advanced users needs or keeping up with the
Joneses in that respect and everyone who wishes a modern programming
environment for designing/implementing bleeding edge statistical
analytical
techniques should learn Python and require the consumers of such
developments to install and maintain python.  Since only about 1 percent
of
SPSS users could care less about such capabilities such considerations
shall
be deferred for the foreseeable future ;-)
---
Maybe a middle ground?  Place certain useful functionality such as various
useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc)
into
a callable library and open up MATRIX to be able to call these functions.
That would NOT require a complete rewrite.


Jon K Peck wrote
Well, MATRIX could use a lot of enhancements, but I will tell you that
using numpy via Python programmability is a thing of beauty, and it has
all the error handling, modularity, expressive power, and functionality
that any user of such a procedure could want.  Here's the top level of
the
reference manual:
http://docs.scipy.org/doc/numpy/reference/index.html#reference

No doubt, the current MATRIX procedure, while powerful, is lacking in a
number of ways, but modernizing it would be a major effort - pretty much
a
rewrite from scratch.  And I am sure that a rewrite would look quite
different from the current syntax, so it would have to be a different
procedure.  When there are free, readily available, and efficient
add-ins
such as numpy, is this really where IBM/SPSS should be spending its
scare
development resources?  It's not a matter of passing the buck: it's an
issue of making the best use of SPSS resources.

My unofficial two cents worth.

Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM

        
[hidden email]

        
new phone: 720-342-5621




From:   David Marso &lt;

        
david.marso@

        
&gt;
To:

        
[hidden email]

        
,
Date:   12/10/2012 07:40 AM
Subject:        [SPSSX-L] The agony and the ecstacy:  Development of
complex              MATRIX              programs.
Sent by:        "SPSSX(r) Discussion" &lt;

        
[hidden email]

        
&gt;



This is not so much a question as much as a general comment and I am
interested in other people's experiences.
Context:
SPSS has had an embedded MATRIX language for over 20 years.
For those with the necessary mathematical background and special needs
it
is
a wonderful tool.
However:
The documentation SUCKS!
Error handling is NON EXISTENT so debugging anything more than trivial
programs is obscenely difficult.
There have been absolutely NO enhancements in the past 20 years.
There is no concept of modularity (no support for subroutines/local
variables).
I would like to see IBM/SPSS development open up the can of worms called
MATRIX and reexamine its potential for developers to create specialized
solutions.
JoNoH... Please don't tell me to use Python.  That would be passing the
buck.
Don't tell me to use R " "...
Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years
behind
the game.
----






-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to
email me.
--
View this message in context:

http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to

        
[hidden email]

        
 (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD



-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to
email me.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to

      
[hidden email]

      
 (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD



-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716829.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD



Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

David Marso
Administrator
I don't know if I would call it 'angry'.  It seems historically that there has been a sort of uneasy coexistence between two distinct factions of SPSS users.  On one hand there is the "Real Stats Real Easy" menu clicking What's a manual" contingent and the "Hard Core SPSS nothing is impossible just give me the tools"  minority and a vast middle ground of "just give me the dog food and I'll eat it".-   When it comes down to the economics of development the majority wins out every time and the needs of the minority bleeding edge "to go where no one has gone before" have been relegated to a 'let them eat python' rather than enhance functional familiar tools.   My client base is primarily academic researchers who contract with me to implement new statistical tools which will be distributed to a diverse user base.
To require their entire distribution to install Python is not an acceptable prerequisite.  Hence my solutions rely upon MACRO and MATRIX (sometimes internal Basic scripting) .. ie complete solution without any external dependencies.  I have developed a set of techniques and code design which have facilitated modular MATRIX programming and some level of debugging but it is a far call from what I have advocated in my post.
 For those who could care here is an insight :  Segment the MATRIX code into logically/functionally distinct modules.  Wrap each module in a DEFINE !ENDDEFINE block.  At the head of each DEFINE block add PRINT /TITLE="module name" .  This approach has saved me what remaining hair I have left after all these years.  It also allows one to 'compile' individual macros independently.  One solution I am currently working on is roughly 1000 lines of code.  Each of the 'MATRIX modules' spans between 50 and 100 lines.  Several modules are used more than once over the overall solution ;-))
Hope this helps anyone who is struggling with building large MATRIX/MACRO solutions .

Kirill Orlov wrote
Again, I can't help agreeing with almost every David's angry lunge.
Because it is all just, and so long sore.

And macros, too. The idea of a macro is basically beautiful: a language
that generates code. It *can* be developed and enhanced, I believe, if
development is willing and thoughtful, not lazy or urgent.

SPSS development has become - to my subjective feeling - so much
negligent that they didn't allowed using GPL within macros - just
because GPL doesn't support quoting with apostrophes (which macro
function !quote does). Also, SPSS stopped developing Chart Builder
(which yet never borrowed some really good features of Interactive
Graphs dialog box); instead, they added yet a new feature, Graphboard
Template. Now we have to wander between the Builder and the Template.
Etc etc. One can continue pulling examples of this compoundness without
unity, a characteristic of modern SPSS Statistics.

10.12.2012 20:54, David Marso ?????:
> You are perhaps too close to the vat of Python Kool Aid...
> The consistent paradigm seems to be to defer to Python to do the heavy
> lifting and hence offload the economics of maintaining and or enhancing
> traditional methods of custom implementation from internal developers to
> python extensions... How long before SPSS itself is merely a collection of
> python extensions?
> Is the internal design of MATRIX so intractible so as to make calling
> external libraries prohibited or providing for localized subroutines/local
> variables?
> Is the internal design of the SPSS core transformation funstionality in such
> a disarray that coordinating a callable library of COMPUTE functions
> constitutes a herculean undertaking?
> MACRO is deprecated!  Why is that so?  Why should it be necessary to resort
> to external development tools rather than have those tools available in the
> SPSS application itself?
>
>
>
> Jon K Peck wrote
>> I disagree completely with Marso's so-called "translation".
>>
>>
>> Jon Peck (no "h") aka Kim
>> Senior Software Engineer, IBM
>> peck@.ibm
>> new phone: 720-342-5621
>>
>>
>>
>>
>> From:   David Marso <
>> david.marso@
>> >
>> To:
>> SPSSX-L@.uga
>> ,
>> Date:   12/10/2012 09:03 AM
>> Subject:        Re: [SPSSX-L] The agony and the ecstasy:  Development of
>> complex              MATRIX              programs.
>> Sent by:        "SPSSX(r) Discussion" <
>> SPSSX-L@.uga
>> >
>>
>>
>>
>> Translation:
>> IBS/SPSS could care less about advanced users needs or keeping up with the
>> Joneses in that respect and everyone who wishes a modern programming
>> environment for designing/implementing bleeding edge statistical
>> analytical
>> techniques should learn Python and require the consumers of such
>> developments to install and maintain python.  Since only about 1 percent
>> of
>> SPSS users could care less about such capabilities such considerations
>> shall
>> be deferred for the foreseeable future ;-)
>> ---
>> Maybe a middle ground?  Place certain useful functionality such as various
>> useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc)
>> into
>> a callable library and open up MATRIX to be able to call these functions.
>> That would NOT require a complete rewrite.
>>
>>
>> Jon K Peck wrote
>>> Well, MATRIX could use a lot of enhancements, but I will tell you that
>>> using numpy via Python programmability is a thing of beauty, and it has
>>> all the error handling, modularity, expressive power, and functionality
>>> that any user of such a procedure could want.  Here's the top level of
>> the
>>> reference manual:
>>> http://docs.scipy.org/doc/numpy/reference/index.html#reference
>>>
>>> No doubt, the current MATRIX procedure, while powerful, is lacking in a
>>> number of ways, but modernizing it would be a major effort - pretty much
>> a
>>> rewrite from scratch.  And I am sure that a rewrite would look quite
>>> different from the current syntax, so it would have to be a different
>>> procedure.  When there are free, readily available, and efficient
>> add-ins
>>> such as numpy, is this really where IBM/SPSS should be spending its
>> scare
>>> development resources?  It's not a matter of passing the buck: it's an
>>> issue of making the best use of SPSS resources.
>>>
>>> My unofficial two cents worth.
>>>
>>> Jon Peck (no "h") aka Kim
>>> Senior Software Engineer, IBM
>>> peck@.ibm
>>> new phone: 720-342-5621
>>>
>>>
>>>
>>>
>>> From:   David Marso <
>>> david.marso@
>>> >
>>> To:
>>> SPSSX-L@.uga
>>> ,
>>> Date:   12/10/2012 07:40 AM
>>> Subject:        [SPSSX-L] The agony and the ecstacy:  Development of
>>> complex              MATRIX              programs.
>>> Sent by:        "SPSSX(r) Discussion" <
>>> SPSSX-L@.uga
>>> >
>>>
>>>
>>>
>>> This is not so much a question as much as a general comment and I am
>>> interested in other people's experiences.
>>> Context:
>>> SPSS has had an embedded MATRIX language for over 20 years.
>>> For those with the necessary mathematical background and special needs
>> it
>>> is
>>> a wonderful tool.
>>> However:
>>> The documentation SUCKS!
>>> Error handling is NON EXISTENT so debugging anything more than trivial
>>> programs is obscenely difficult.
>>> There have been absolutely NO enhancements in the past 20 years.
>>> There is no concept of modularity (no support for subroutines/local
>>> variables).
>>> I would like to see IBM/SPSS development open up the can of worms called
>>> MATRIX and reexamine its potential for developers to create specialized
>>> solutions.
>>> JoNoH... Please don't tell me to use Python.  That would be passing the
>>> buck.
>>> Don't tell me to use R " "...
>>> Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years
>> behind
>>> the game.
>>> ----
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----
>>> Please reply to the list and not to my personal email.
>>> Those desiring my consulting or training services please feel free to
>>> email me.
>>> --
>>> View this message in context:
>>>
>> http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
>>
>>> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>>>
>>> =====================
>>> To manage your subscription to SPSSX-L, send a message to
>>> LISTSERV@.UGA
>>>   (not to SPSSX-L), with no body text except the
>>> command. To leave the list, send the command
>>> SIGNOFF SPSSX-L
>>> For a list of commands to manage subscriptions, send the command
>>> INFO REFCARD
>>
>>
>>
>> -----
>> Please reply to the list and not to my personal email.
>> Those desiring my consulting or training services please feel free to
>> email me.
>> --
>> View this message in context:
>> http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html
>>
>> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>> LISTSERV@.UGA
>>   (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to email me.
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716829.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

David Marso
Administrator
There were a number of off list requests for an example of my development
technique WRT MATRIX and MACRO.
Here is the simplest possible example.  One can imagine the utility when the code begins to span some 20 pages of uber tight MATRIX code and conditional logic / looping and other madness ;-)
Look ma, I still have hair!!!
----
data list free / x y.
begin data
1 3 2 6 3 5 6 1 3 6 1 5 6 3 5 1 6 7 3 5 1 2 6 3 5 1 6 3 5 6 1 3 1 2 5 7
end data.


DEFINE MATALL ().
MATRIX .
GET x / file * / var x.
GET y / file * / var y.
COMPUTE NR=NROW(X).
COMPUTE one=MAKE(NR,1,1).
COMPUTE augX={one,X}.
COMPUTE b=GINV(T(augX)*augX)*T(augX)*Y.
COMPUTE pred=augX*b.
COMPUTE resid=Y-Pred.
COMPUTE SSE=SSCP(resid).
COMPUTE SST=SSCP(y).
COMPUTE SSP=SST-SSE.
PRINT b.
PRINT {SST;SSP;SSE}.
END MATRIX.
!ENDDEFINE.
MATALL.



DEFINE GETDATA ().
DO IF (debug GT 0).
PRINT / TITLE "getdata".
END IF.
GET x / file * / var x.
GET y / file * / var y.
!ENDDEFINE.

DEFINE COMP () .
DO IF (debug GT 0).
PRINT / TITLE "comp".
END IF.
COMPUTE NR=NROW(X).
COMPUTE one=MAKE(NR,1,1).
COMPUTE augX={one,X}.
COMPUTE b=GINV(T(augX)*augX)*T(augX)*Y.
COMPUTE pred=augX*b.
COMPUTE resid=Y-Pred.
COMPUTE SSE=SSCP(resid).
COMPUTE SST=SSCP(y).
COMPUTE SSP=SST-SSE.
!ENDDEFINE.

DEFINE REP ().
DO IF (debug GT 0).
PRINT / TITLE "rep".
END IF.
PRINT b.
PRINT {SST;SSP;SSE}.
!ENDDEFINE .

DEFINE MATMAIN ( DEBUG !TOKENS(1) !DEFAULT (0)).
MATRIX.
COMPUTE debug=!DEBUG.
GETDATA .
COMP .
REP .
END MATRIX.
!ENDDEFINE.

MATMAIN .
MATMAIN DEBUG 1.

David Marso wrote
I don't know if I would call it 'angry'.  It seems historically that there has been a sort of uneasy coexistence between two distinct factions of SPSS users.  On one hand there is the "Real Stats Real Easy" menu clicking What's a manual" contingent and the "Hard Core SPSS nothing is impossible just give me the tools"  minority and a vast middle ground of "just give me the dog food and I'll eat it".-   When it comes down to the economics of development the majority wins out every time and the needs of the minority bleeding edge "to go where no one has gone before" have been relegated to a 'let them eat python' rather than enhance functional familiar tools.   My client base is primarily academic researchers who contract with me to implement new statistical tools which will be distributed to a diverse user base.
To require their entire distribution to install Python is not an acceptable prerequisite.  Hence my solutions rely upon MACRO and MATRIX (sometimes internal Basic scripting) .. ie complete solution without any external dependencies.  I have developed a set of techniques and code design which have facilitated modular MATRIX programming and some level of debugging but it is a far call from what I have advocated in my post.
 For those who could care here is an insight :  Segment the MATRIX code into logically/functionally distinct modules.  Wrap each module in a DEFINE !ENDDEFINE block.  At the head of each DEFINE block add PRINT /TITLE="module name" .  This approach has saved me what remaining hair I have left after all these years.  It also allows one to 'compile' individual macros independently.  One solution I am currently working on is roughly 1000 lines of code.  Each of the 'MATRIX modules' spans between 50 and 100 lines.  Several modules are used more than once over the overall solution ;-))
Hope this helps anyone who is struggling with building large MATRIX/MACRO solutions .

Kirill Orlov wrote
Again, I can't help agreeing with almost every David's angry lunge.
Because it is all just, and so long sore.

And macros, too. The idea of a macro is basically beautiful: a language
that generates code. It *can* be developed and enhanced, I believe, if
development is willing and thoughtful, not lazy or urgent.

SPSS development has become - to my subjective feeling - so much
negligent that they didn't allowed using GPL within macros - just
because GPL doesn't support quoting with apostrophes (which macro
function !quote does). Also, SPSS stopped developing Chart Builder
(which yet never borrowed some really good features of Interactive
Graphs dialog box); instead, they added yet a new feature, Graphboard
Template. Now we have to wander between the Builder and the Template.
Etc etc. One can continue pulling examples of this compoundness without
unity, a characteristic of modern SPSS Statistics.

10.12.2012 20:54, David Marso ?????:
> You are perhaps too close to the vat of Python Kool Aid...
> The consistent paradigm seems to be to defer to Python to do the heavy
> lifting and hence offload the economics of maintaining and or enhancing
> traditional methods of custom implementation from internal developers to
> python extensions... How long before SPSS itself is merely a collection of
> python extensions?
> Is the internal design of MATRIX so intractible so as to make calling
> external libraries prohibited or providing for localized subroutines/local
> variables?
> Is the internal design of the SPSS core transformation funstionality in such
> a disarray that coordinating a callable library of COMPUTE functions
> constitutes a herculean undertaking?
> MACRO is deprecated!  Why is that so?  Why should it be necessary to resort
> to external development tools rather than have those tools available in the
> SPSS application itself?
>
>
>
> Jon K Peck wrote
>> I disagree completely with Marso's so-called "translation".
>>
>>
>> Jon Peck (no "h") aka Kim
>> Senior Software Engineer, IBM
>> peck@.ibm
>> new phone: 720-342-5621
>>
>>
>>
>>
>> From:   David Marso <
>> david.marso@
>> >
>> To:
>> SPSSX-L@.uga
>> ,
>> Date:   12/10/2012 09:03 AM
>> Subject:        Re: [SPSSX-L] The agony and the ecstasy:  Development of
>> complex              MATRIX              programs.
>> Sent by:        "SPSSX(r) Discussion" <
>> SPSSX-L@.uga
>> >
>>
>>
>>
>> Translation:
>> IBS/SPSS could care less about advanced users needs or keeping up with the
>> Joneses in that respect and everyone who wishes a modern programming
>> environment for designing/implementing bleeding edge statistical
>> analytical
>> techniques should learn Python and require the consumers of such
>> developments to install and maintain python.  Since only about 1 percent
>> of
>> SPSS users could care less about such capabilities such considerations
>> shall
>> be deferred for the foreseeable future ;-)
>> ---
>> Maybe a middle ground?  Place certain useful functionality such as various
>> useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc)
>> into
>> a callable library and open up MATRIX to be able to call these functions.
>> That would NOT require a complete rewrite.
>>
>>
>> Jon K Peck wrote
>>> Well, MATRIX could use a lot of enhancements, but I will tell you that
>>> using numpy via Python programmability is a thing of beauty, and it has
>>> all the error handling, modularity, expressive power, and functionality
>>> that any user of such a procedure could want.  Here's the top level of
>> the
>>> reference manual:
>>> http://docs.scipy.org/doc/numpy/reference/index.html#reference
>>>
>>> No doubt, the current MATRIX procedure, while powerful, is lacking in a
>>> number of ways, but modernizing it would be a major effort - pretty much
>> a
>>> rewrite from scratch.  And I am sure that a rewrite would look quite
>>> different from the current syntax, so it would have to be a different
>>> procedure.  When there are free, readily available, and efficient
>> add-ins
>>> such as numpy, is this really where IBM/SPSS should be spending its
>> scare
>>> development resources?  It's not a matter of passing the buck: it's an
>>> issue of making the best use of SPSS resources.
>>>
>>> My unofficial two cents worth.
>>>
>>> Jon Peck (no "h") aka Kim
>>> Senior Software Engineer, IBM
>>> peck@.ibm
>>> new phone: 720-342-5621
>>>
>>>
>>>
>>>
>>> From:   David Marso <
>>> david.marso@
>>> >
>>> To:
>>> SPSSX-L@.uga
>>> ,
>>> Date:   12/10/2012 07:40 AM
>>> Subject:        [SPSSX-L] The agony and the ecstacy:  Development of
>>> complex              MATRIX              programs.
>>> Sent by:        "SPSSX(r) Discussion" <
>>> SPSSX-L@.uga
>>> >
>>>
>>>
>>>
>>> This is not so much a question as much as a general comment and I am
>>> interested in other people's experiences.
>>> Context:
>>> SPSS has had an embedded MATRIX language for over 20 years.
>>> For those with the necessary mathematical background and special needs
>> it
>>> is
>>> a wonderful tool.
>>> However:
>>> The documentation SUCKS!
>>> Error handling is NON EXISTENT so debugging anything more than trivial
>>> programs is obscenely difficult.
>>> There have been absolutely NO enhancements in the past 20 years.
>>> There is no concept of modularity (no support for subroutines/local
>>> variables).
>>> I would like to see IBM/SPSS development open up the can of worms called
>>> MATRIX and reexamine its potential for developers to create specialized
>>> solutions.
>>> JoNoH... Please don't tell me to use Python.  That would be passing the
>>> buck.
>>> Don't tell me to use R " "...
>>> Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years
>> behind
>>> the game.
>>> ----
>>>
>>>
>>>
>>>
>>>
>>>
>>> -----
>>> Please reply to the list and not to my personal email.
>>> Those desiring my consulting or training services please feel free to
>>> email me.
>>> --
>>> View this message in context:
>>>
>> http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html
>>
>>> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>>>
>>> =====================
>>> To manage your subscription to SPSSX-L, send a message to
>>> LISTSERV@.UGA
>>>   (not to SPSSX-L), with no body text except the
>>> command. To leave the list, send the command
>>> SIGNOFF SPSSX-L
>>> For a list of commands to manage subscriptions, send the command
>>> INFO REFCARD
>>
>>
>>
>> -----
>> Please reply to the list and not to my personal email.
>> Those desiring my consulting or training services please feel free to
>> email me.
>> --
>> View this message in context:
>> http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html
>>
>> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>> LISTSERV@.UGA
>>   (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to email me.
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716829.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Zuluaga, Juan
In reply to this post by Jon K Peck
David, please take a look at R or NumPy.
IMHO it's easier to write (and read) algorithms in them.
Their growth has been driven by skilled users (people like you) who create packages out of their own code.
Perhaps IBM/SPSS has realized that they can't compete for that type of user.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Bruce Weaver
Administrator
From one of David's posts in this thread:

"My client base is primarily academic researchers who contract with me to implement new statistical tools which will be distributed to a diverse user base.  To require their entire distribution to install Python is not an acceptable prerequisite.  Hence my solutions rely upon MACRO and MATRIX (sometimes internal Basic scripting) .. ie complete solution without any external dependencies."

This describes to a tee virtually all of the academic users I know.  For some reason, academic users seem to think that it should be possible to get the job done using SPSS alone.  ;-)


Zuluaga, Juan wrote
David, please take a look at R or NumPy.
IMHO it's easier to write (and read) algorithms in them.
Their growth has been driven by skilled users (people like you) who create packages out of their own code.
Perhaps IBM/SPSS has realized that they can't compete for that type of user.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Ruben Geert van den Berg
But is it relevant whether "it's possible to get the job done using SPSS alone"? I mean, it's also possible for me to walk to China.

I think a more relevant question is how to get things done with minimal effort and maximal precision. And that often involves Python.

Now if Python is open source, then why wouldn't IBM embed its source code into SPSS?

Then all this functionality would be available to everybody "without any external dependencies".

Everybody happy!

Ruben

> Date: Mon, 10 Dec 2012 18:20:37 -0800

> From: [hidden email]
> Subject: Re: The agony and the ecstasy: Development of complex MATRIX programs.
> To: [hidden email]
>
> From one of David's posts in this thread:
>
> "My client base is primarily academic researchers who contract with me to
> implement new statistical tools which will be distributed to a diverse user
> base. To require their entire distribution to install Python is not an
> acceptable prerequisite. Hence my solutions rely upon MACRO and MATRIX
> (sometimes internal Basic scripting) .. ie complete solution without any
> external dependencies."
>
> This describes to a tee virtually all of the academic users I know. For
> some reason, academic users seem to think that it should be possible to get
> the job done using SPSS alone. ;-)
>
>
>
> Zuluaga, Juan wrote
> > David, please take a look at R or NumPy.
> > IMHO it's easier to write (and read) algorithms in them.
> > Their growth has been driven by skilled users (people like you) who create
> > packages out of their own code.
> > Perhaps IBM/SPSS has realized that they can't compete for that type of
> > user.
> >
> > =====================
> > To manage your subscription to SPSSX-L, send a message to
>
> > LISTSERV@.UGA
>
> > (not to SPSSX-L), with no body text except the
> > command. To leave the list, send the command
> > SIGNOFF SPSSX-L
> > For a list of commands to manage subscriptions, send the command
> > INFO REFCARD
>
>
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
>
> "When all else fails, RTFM."
>
> NOTE: My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716855.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Marta Garcia-Granero
In reply to this post by Kirill Orlov
Hi Kiril, long time

El 10/12/2012 18:39, Kirill Orlov escribió:
... SPSS development has become - to my subjective feeling - so much negligent that they didn't allowed using GPL within macros - just because GPL doesn't support quoting with apostrophes (which macro function !quote does). ....
But I DO have a macro with GPL code inside, thanks to ViAnn, who helped a lot with the quote-unquote-double quote-single quote waltz needed. Maybe it is another example of the bunblebee flying although it should not be possible (http://www.sciencenews.org/view/generic/id/5400/title/Math_Trek__Flight_of_the_Bumblebee).

Here is the macro, just in case it helps you.

BTW, fiddling with my old macros is like looking at an old photo album, full of faded pictures...

Warmest regards,
Marta GG

*****************************************************************
*   KALBFLEISCH-PRENTICE 95% CI INTERVALS FOR SURVIVAL CURVES   *
*****************************************************************
* Kalbfleisch JD and Prentice RL(1980). The Statistical         *
* Analysis of Failure Time Data. New York: John Wiley & Sons    *
*****************************************************************
* (c) Marta Garcia-Granero (june 2005 - original code) modified *
* (turnt into macro) september 2008.  [hidden email]  *
*****************************************************************
         * Downloaded from: http://gjyp.nl/marta/      *
         * Feel free to use or modify this code, but   *
         * acknowledge the author and the web site     *
         ***********************************************

* Unlike Greenwood's, the Kalbfleisch–Prentice approach has the attractive
  property that the upper and lower bounds are always between 0 and 1.

* MACRO definition *.
DEFINE KPSURVIVAL(!POS=!TOKENS(1)/!POS=!CHAREND('(')/!POS=!CHAREND(')')).
* Creating the chart template *.
PRESERVE.
SET RESULTS=NONE MESSAGES=NONE ERRORS=NONE.
HOST COMMAND=['IF not exist C:\Temp MD C:\Temp'].
HOST COMMAND=['DEL C:\Temp\*.sgt'].
RESTORE.
DO IF $casenum EQ 1.
* Dynamically create the template for the chart *.
WRITE OUTFILE='C:\Temp\KPTemplate.sgt'
/'<?xml version="1.0" encoding="UTF-8" standalone="no"?>'
/'<template SPSS-Version="2.2" selectPath="12 47 "'
/' xmlns="http://xml.spss.com/spss/visualization"'
/' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"'
/' xsi:schemaLocation="http://xml.spss.com/spss/visualization'
/' http://xml.spss.com/spss/visualization/vizml-template-3.0.xsd">'
/'    <setAxisMargin categorical="false" lowerMargin="0%" role="x" upperMargin="5%"/>'
/'    <setAxisMargin categorical="false" lowerMargin="0%" role="y" upperMargin="5%"/>'
/'    <addFrame count="1" styleOnly="true" type="visualization">'
/'        <style color="#ffffff" color2="transparent" number="0" visible="true"/>'
/'        <style font-family="SansSerif" font-size="8pt" number="1" pattern="0" stroke-linecap="butt" text-fit="true" visible="true"/>'
/'    </addFrame>'
/'    <addFrame count="1" styleOnly="true" type="graph">'
/'        <style color="transparent" color2="transparent" visible="true"/>'
/'        <style color="#ffffff" color2="transparent" number="1" visible="true"/>'
/'    </addFrame>'
/'    <addFrame count="1" styleOnly="true" type="title">'
/'        <style color="transparent" color2="transparent" visible="true"/>'
/'    </addFrame>'
/'</template>'.
END IF.
EXECUTE.
DATASET NAME OriginalData.
* Copy data to leave original unmodified *.
DATASET COPY WorkingData1 WINDOW=MINIMIZED.
DATASET DECLARE AuxData WINDOW=HIDDEN.
DATASET ACTIVATE WorkingData1.
COMPUTE status=(!2 EQ !3).
RENAME VARIABLES (!1 = time).
* Get rid of every other variable *.
MATCH FILES FILE=*
 /KEEP=time status.
* Save sorted file to match CI limits later *.
SORT CASES BY time(A).
DATASET COPY WorkingData2  WINDOW=MINIMIZED.
* Saving KM survival estimates *.
PRESERVE.
SET ERRORS=NONE RESULTS=NONE.
KM time /STATUS=status(1)
  /PRINT NONE
  /PLOT SURVIVAL
  /SAVE SURVIVAL  .
* Total sample size *.
RANK VARIABLES=time(A) /N INTO totaln/PRINT=NO.
* Preparing data to compute pj and Std Error of Log(survival) *.
AGGREGATE /OUTFILE = *  MODE=REPLACE
  /BREAK = time
  /aj = SUM(status)
  /sj = FIRST(SUR_1)
  /totaln = FIRST(totaln)
  /N = n.
* Auxiliary file #1 *.
MATRIX.
COMPUTE time=0.
COMPUTE aj=0.
COMPUTE sj=1.
COMPUTE totaln=0.
COMPUTE n=0.
COMPUTE vnames={'time','aj','sj','totaln','n'}.
COMPUTE newdata={time,aj,sj,totaln,n}.
SAVE newdata /OUTFILE='AuxData'/NAMES=vnames.
END MATRIX.
* Adding time 0 data (from auxiliary file)*.
ADD FILES /FILE=*
 /FILE='AuxData'
 /IN=source.
IF source EQ 1 totaln=LAG(totaln).
SORT CASES BY time(A).
* Computing auxiliary statistics *.
COMPUTE rj=totaln.
DO IF $casenum GT 1.
- COMPUTE n = n+LAG(n) .
- COMPUTE rj= totaln-LAG(n).
END IF.
EXECUTE ./* Don't eliminate it (necessary) *.
* Getting rid of censored data *.
SELECT IF NOT MISSING(sj).
* More statistics and reordering dataset *.
COMPUTE j= $casenum-1.
COMPUTE pj = (rj-aj)/rj .
MATCH FILES FILE = *
 /KEEP = j time aj rj pj sj .
* 95% CI limits *.
COMPUTE sum = ((1-pj)/(pj*rj)).
DO IF $casenum GT 1.
- COMPUTE sum=sum+LAG(sum).
- COMPUTE sdlog = SQRT(sum/((LN(sj))**2)).
* 95% Limits using Log(-Log(Sj)) & back transform
* Replace 1.959964 by 2.575829 if 99%CI are wanted *.
- COMPUTE lower = EXP(-EXP(LN(-LN(sj))+1.959964*sdlog)).
- COMPUTE upper = EXP(-EXP(LN(-LN(sj))-1.959964*sdlog)).
END IF.
RESTORE.
* Report *.
FORMAT j TO rj (F8.0) sj pj lower upper (F8.3).
SUMMARIZE
 /TABLES=time sj lower upper
 /FORMAT=LIST NOCASENUM NOTOTAL
 /TITLE='Kaplan-Meier Estimates and Kalbfleisch-Prentice 95% CI'
 /CELLS=NONE.
* Adding survival & CI limits to original dataset *.
MATCH FILES FILE=*
 /KEEP = time sj lower upper.
MATCH FILES
 /FILE='WorkingData2'
 /TABLE=*
 /BY time.
* Adding time=0 survival data (auxiliary file #2) *.
PRESERVE.
SET ERRORS=NONE RESULTS=NONE.
MATRIX.
COMPUTE time=0.
COMPUTE sj=1.
COMPUTE lower=1.
COMPUTE upper=1.
COMPUTE vnames={'time','sj','lower','upper'}.
COMPUTE newdata={time,sj,lower,upper}.
SAVE newdata /OUTFILE='AuxData'/NAMES=vnames.
END MATRIX.
ADD FILES /FILE=*
 /FILE='AuxData'.
SORT CASES BY time(a).
* Filling data for censored cases after the first event *.
DO IF MISSING(sj).
- COMPUTE sj=LAG(sj).
- COMPUTE lower=LAG(lower).
- COMPUTE upper=LAG(upper).
END IF.
RESTORE.
* Survival plot (fully formatted, thanks to ViAnn Beadle) *.
FORMAT sj lower upper (F8.1).
!LET !gdataset=!CONCAT('"','graphdataset','"').
!LET !time=!CONCAT('"','time','"').
!LET !sj=!CONCAT('"','sj','"').
!LET !lower=!CONCAT('"','lower','"').
!LET !upper=!CONCAT('"','upper','"').
!LET !ylabel=!CONCAT('"','Cumulative Survival','"').
!LET !xlabel=!CONCAT('"','Follow-up Time','"').
!LET !title=!CONCAT('"','KM estimate with Kalbfleish-Prentice 95%CI','"').
!LET !px=!CONCAT('"','2px','"').
GGRAPH
  /GRAPHDATASET NAME=!gdataset VARIABLES=time sj lower upper
  MISSING=LISTWISE REPORTMISSING=NO
  /GRAPHSPEC SOURCE=INLINE TEMPLATE="C:\Temp\KPTemplate.sgt".
BEGIN GPL
  SOURCE: s=userSource(id(!gdataset))
  DATA: time=col(source(s), name(!time))
  DATA: sj=col(source(s), name(!sj))
  DATA: lower=col(source(s), name(!lower))
  DATA: upper=col(source(s), name(!upper))
  SCALE: linear(dim(2), include(0))
  GUIDE: axis(dim(1), label(!xlabel))
  GUIDE: axis(dim(2), label(!ylabel), delta(0.1))
  GUIDE: legend(aesthetic(aesthetic.shape.interior), null())
  GUIDE: text.title(label(!title))
  ELEMENT: line(position(smooth.step.left(time*lower)), missing.wings(),
           shape.interior(shape.dash), color(color.black))
  ELEMENT: line(position(smooth.step.left(time*upper)), missing.wings(),
           shape.interior(shape.dash), color(color.black))
  ELEMENT: line(position(smooth.step.left(time*sj)), missing.wings(),
           shape.interior(shape.solid), size(size.!px))
END GPL.
* Tidying up *.
DATASET ACTIVATE OriginalData WINDOW=ASIS.
DATASET CLOSE AuxData.
DATASET CLOSE WorkingData1.
DATASET CLOSE WorkingData2.
!ENDDEFINE.


*****************************************************************
*                     EXAMPLE DATASET                           *
* Table 9.1 Biostatistical Methods in Epidemiology (pp 177-178) *
* Stephen C Newman (2001); JOHN WILEY&SONS                      *
*****************************************************************.
DATA LIST FREE/FuTime Event stage reclevel (4 F8.0).
BEGIN DATA
50 0 1 1 51 1 1 1 51 0 1 1 53 0 1 1 53 0 1 1 54 0 1 1
54 0 1 1 55 0 1 1 56 1 1 1 56 0 1 1 57 0 1 1 60 0 1 1
10 1 1 2 34 1 1 2 34 0 1 2 47 1 1 2 47 1 1 2 49 0 1 2
49 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2
50 0 1 2 50 0 1 2 51 0 1 2 51 0 1 2 51 0 1 2 51 0 1 2
51 0 1 2 51 0 1 2 52 0 1 2 52 0 1 2 52 0 1 2 52 0 1 2
52 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2
53 0 1 2 54 0 1 2 54 0 1 2 54 0 1 2 54 0 1 2 54 0 1 2
55 0 1 2 55 0 1 2 56 0 1 2 56 0 1 2 57 0 1 2 57 0 1 2
57 0 1 2 57 0 1 2 57 0 1 2 58 0 1 2 58 0 1 2 58 0 1 2
58 0 1 2 58 0 1 2 59 0 1 2 59 0 1 2 59 0 1 2 59 0 1 2
60 0 1 2 60 0 1 2 60 0 1 2  4 0 2 1  9 1 2 1 13 1 2 1
21 1 2 1 29 1 2 1 29 1 2 1 40 1 2 1 46 1 2 1 49 0 2 1
49 0 2 1 52 0 2 1 52 0 2 1 53 0 2 1 54 0 2 1 55 0 2 1
55 0 2 1 56 0 2 1 57 1 2 1 57 0 2 1 58 0 2 1 58 0 2 1
59 0 2 1 60 0 2 1 11 1 2 2 16 1 2 2 21 1 2 2 23 1 2 2
23 1 2 2 24 1 2 2 33 1 2 2 33 1 2 2 36 1 2 2 36 1 2 2
36 0 2 2 37 1 2 2 45 1 2 2 46 1 2 2 49 0 2 2 49 0 2 2
50 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2
51 0 2 2 51 0 2 2 51 0 2 2 51 0 2 2 52 0 2 2 52 0 2 2
52 0 2 2 52 0 2 2 52 0 2 2 53 0 2 2 53 0 2 2 53 0 2 2
53 0 2 2 53 0 2 2 54 0 2 2 54 0 2 2 54 0 2 2 54 0 2 2
55 0 2 2 55 0 2 2 55 0 2 2 55 0 2 2 56 0 2 2 56 0 2 2
56 0 2 2 56 0 2 2 56 0 2 2 56 0 2 2 57 0 2 2 57 0 2 2
57 0 2 2 57 0 2 2 58 1 2 2 58 1 2 2 58 0 2 2 58 0 2 2
58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2
59 0 2 2 59 0 2 2 59 0 2 2 59 0 2 2 59 0 2 2 60 0 2 2
60 0 2 2 60 0 2 2 60 0 2 2 60 0 2 2 60 0 2 2  9 1 3 1
12 1 3 1 14 1 3 1 15 1 3 1 15 0 3 1 17 1 3 1 21 1 3 1
22 1 3 1 23 1 3 1 23 1 3 1 31 1 3 1 34 1 3 1 35 1 3 1
53 0 3 1 60 0 3 1  7 0 3 2  9 1 3 2 17 1 3 2 21 0 3 2
22 1 3 2 22 1 3 2 34 1 3 2 34 1 3 2 41 1 3 2 49 0 3 2
52 0 3 2 55 1 3 2 56 0 3 2 58 0 3 2 58 0 3 2 59 0 3 2
59 0 3 2
END DATA.
VAR LABEL stage' Tumour Stage'/ reclevel 'Receptor level'
/FuTime 'Survival time (months)' /Event 'Censoring Status'.
VALUE label stage 1'I' 2'II' 3'III'
 /reclevel 1'Low' 2'High' /Event 0'Censored' 1'Event'.

KPSURVIVAL FuTime Event(1).

Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Kirill Orlov
In reply to this post by Ruben Geert van den Berg
Ruben, you entirely forget of a subset of users who are rather creative individuals than individual consumers. Those ones would defy "doing things with minimal effort". They ultimately dream to rewrite all the libraries existent in Python or Matlab now in SPSS native syntax. They think that SPSS deserves going its own proud way and to have its own community of users-developers, not Python's community.

And now that Python *is* already in SPSS, people who like Python - the rational "consumers" - should feel happy and settled (nobody votes to delete Python or R back from SPSS, it'd be idiotic). Why not now to satisfy the others - the "creatives" nuts - and to give them a bit enriched programmability tools within old macro and matrix facilities, just because these people rely on them and don't like Python or R?


11.12.2012 9:26, Ruben van den Berg пишет:
But is it relevant whether "it's possible to get the job done using SPSS alone"? I mean, it's also possible for me to walk to China.

I think a more relevant question is how to get things done with minimal effort and maximal precision. And that often involves Python.

Now if Python is open source, then why wouldn't IBM embed its source code into SPSS?

Then all this functionality would be available to everybody "without any external dependencies".

Everybody happy!

Ruben


Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

Kirill Orlov
In reply to this post by Marta Garcia-Granero
Hello Marta, yeah long time to see.

Yes, it is possible to perform amusing pas like
!LET !gdataset=!CONCAT('"','graphdataset','"').
But it is outside of GPL block. It looks so awkward that one should feel himself uneasy.

The GPL block *must* itself permit using macro functions and macro statements inside itself - like any usual syntax does permit it. But somebody in SPSS company is too lazy *to get inside GPL and overhaul some aspects of it*. Not they. They prefer to take GPL "as is" even if it can't get on smoothly with SPSS old residents such as macros. SPSS team do not esteem macro facility , their own creature.

Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

David Marso
Administrator
In reply to this post by Bruce Weaver
One interesting fact is that a number of my recent projects have already been prototyped in R and as part of the research grants the principals were required to provide a native SPSS programming solution as a supporting implementation (ie publish a new analytical technique and provide a means of interesting consumers to create analyses of their own data sets).  One thing I learned from doing these projects is that it seems almost "natural" to create virtually incomprehensible, bRamaging, retinal hemorrhage inducing code in R .  Somewhat more complicated to do the same in SPSS MATRIX (I must plead guilty to possessing this marvelous ability to one-line some rather scary MATRIX pearls -but despite all my efforts towards deep obsfucation they still maintain a humanly readable character- Of course such eye bleeders are surrounded by ample comments (I already have job security in that regard).  I am not at liberty due to NDA considerations to post any of these R beauties (or SPSS attempts) but some of them rival APL http://langexplr.blogspot.com/2011/03/quick-look-at-apl.html in their ease of parsability  .  Some of the more 'clever?' almost ALWAYS uncommented Python code I have seen posted on this list can vie for a similar claim to fame.  
--
Bruce Weaver wrote
From one of David's posts in this thread:

"My client base is primarily academic researchers who contract with me to implement new statistical tools which will be distributed to a diverse user base.  To require their entire distribution to install Python is not an acceptable prerequisite.  Hence my solutions rely upon MACRO and MATRIX (sometimes internal Basic scripting) .. ie complete solution without any external dependencies."

This describes to a tee virtually all of the academic users I know.  For some reason, academic users seem to think that it should be possible to get the job done using SPSS alone.  ;-)


Zuluaga, Juan wrote
David, please take a look at R or NumPy.
IMHO it's easier to write (and read) algorithms in them.
Their growth has been driven by skilled users (people like you) who create packages out of their own code.
Perhaps IBM/SPSS has realized that they can't compete for that type of user.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: The agony and the ecstasy: Development of complex MATRIX programs.

David Marso
Administrator
In reply to this post by Kirill Orlov
One thing I suspect is that the uber brilliant programmer (I knew the man) who wrote MACRO has retired and everyone else is scared s#!^|<$$ to open it up and poke around lest they rouse a dragon or demon and completely FUBAR the rest of the central system.  Same goes with MATRIX which as I understand it was developed externally from SPSS (from the FM "The MATRIX procedure was originally developed at the Madison Academic Computing Center, University of Wisconsin.".
--
Kirill Orlov wrote
Hello Marta, yeah long time to see.

Yes, it is possible to perform amusing pas like
!LET !gdataset=!CONCAT('"','graphdataset','"').
But it is outside of GPL block. It looks so awkward that one should feel
himself uneasy.

The GPL block *must* itself permit using macro functions and macro
statements inside itself - like any usual syntax does permit it. But
somebody in SPSS company is too lazy *to get inside GPL and overhaul
some aspects of it*. Not they. They prefer to take GPL "as is" even if
it can't get on smoothly with SPSS old residents such as macros. SPSS
team do not esteem macro facility , their own creature.
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
12