[xquery-talk] actually, nobody KNOWS about XQuery ....

[daniela florescu]

Ihe,

you asked why XQuery is not more popular.

Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.

Just look at this example.

https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like <https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like>

I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”

[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some meditation
.]]]

In fact, I know that this is not his/her limitation.

It’s our OWN failure to explain to the world what XQuery is, what it does, and what is good at.

Best regards
Dana


P.S. And after that, please DON’T ask me why I am SO pissed off at MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..

They MADE all their money out of the power of XQuery (expressiveness, productivity, etc), yet they pretend they’ve never heard of it
.

That’s something that REALLY gets me angry.

And this will come back to bait them on the business side very badly too.

Oracle would have NEVER done the same thing about SQL
..just saying.

[Ihe Onwuka]

Post by daniela florescu
Ihe,
you asked why XQuery is not more popular.

I'd rather put it that I asked why people are so resistant to using it for
the domain for which it is optimal (querying semi-structured data).

Post by daniela florescu
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”

This is where I take slight issue with G Ken Holman's post last week on
learning stuff, 99% of which I wholeheartedly agree with.

He said university taught him how to learn not what to learn but I think
going to university should give you an awareness of what you ought to know,
ergo it should have some bearing on what to learn. I did my degree as a
mature student. Before university I would approach a problem by solving it
the tools I already knew, hence I wrote a validation test harness in
VBScript that I later learned could have been encoded as an XML Schema.

I also learned the skill of learning what not to learn. Before university
I would learn a tool only to subsequently reflect that it is not something
that I shouldn't be using. Now I can sense it early on. I bought Norm
Matloff's the Art of R and by chapter 2 it was on it's way back to Amazon
Trade-Ins because it was stark staring obvious to me that this might be a
language for statisticians but it certainly wasn't one for a computer
scientist. This excellent talk pretty much ratifies all my hunches and I am
chuffed that I did not have to discover these things experentially.

It worries me when I hear that the likes of Oracle and (I think) IBM are
going to start pushing this language.

So returning to this chap. My initial instinct is that he is one of the
many statisticians and/or accountants running around telling people how to
program (see previous remark about how IT has presided over it's
amateurisation). Alas his educational background is in computing so what we
have is a well educated fellow who lacks awareness of what he ought to
know.

This should not be that surprising though as people in this industry are
notorious for letting others do their thinking for them - hence
Thoughtworks have an audience for their pontifications about what
technologies should be adopted when and other self appointed guru's
regularly surface on the WMD site taking on subjects way beyond the realm
of their actual expertise to a receptive uncritical audience.

Post by daniela florescu
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some
meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it does,
and what is good at.

For a language that was not meant to be specific to XML it got too tightly
coupled to it's initial concrete actualisation. Now that association is a
liability because of the effectiveness of JSON propaganda which keeps
giving the impression that you don't need any other format to an industry
in which too many people are happy to let others do their thinking for them.

Additionally too many devs are beholden to the Jedi Mind Trick of learning
and adopting language libraries without realising that for the same effort
they could learn a DSL that would be much more powerful tool for the job at
hand (and I don't just mean that wrt to data management).

Post by daniela florescu
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at
MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness,
productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.

Quite possibly. I mentioned before that IBM never bet against SQL when it
was pushing DB2 so it would not surprise me if MarkLogic's stance manifests
as a strategic error.

[daniela florescu]

One thing that did strike me is the link cited in one of the comments about this article:

http://queue.acm.org/detail.cfm?id=1961297 <http://queue.acm.org/detail.cfm?id=1961297>

Just look at the examples.

Does everyone see what I see !? :-)

Those are just FLWOR expressions, with FOR spelled as FROM and RETURN spelled
as SELECT.

Can we PLEASE add those as synonyms in the XQuery grammar before you close XQuery 3.1 !?

Otherwise we’ll hear for another 100 years that XQuery has nothing to do with SQL while THIS language
describe in this paper DOES. (sic!)

It’s dumb, but that’s how it is.

Pretty please !???

Thanks
Dana

Post by daniela florescu
Ihe,
you asked why XQuery is not more popular.
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like <https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like>
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it does, and what is good at.
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness, productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.

[daniela florescu]

Pavel, I hope you don’t mind that I forward your personal email to the main list.

Thanks for the links.

Yes, I am of course aware of the history of it, but I don’t think everyone else is.

(I even gave feedback to Erik Meijer before he submitted this proposal to Bill Gates
. and I lost
a beer with him because we bet who among the two of us knows XML Schema better — and he did :-).

This reinforces the ideas that:

1. the principles of XQuery are orthogonal to XML itself and

2. spelling FROM and SELECT seems to matter to people A LOT.

Best regards
Dana

https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx <https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx>
https://msdn.microsoft.com/en-us/library/ms974195.aspx <https://msdn.microsoft.com/en-us/library/ms974195.aspx>
It became LINQ after it dropped ties to XDM and was generalized to operate on arbitrary CLI object graphs instead.
http://queue.acm.org/detail.cfm?id=1961297 <http://queue.acm.org/detail.cfm?id=1961297>
Just look at the examples.
Does everyone see what I see !? :-)
Those are just FLWOR expressions, with FOR spelled as FROM and RETURN spelled
as SELECT.
Can we PLEASE add those as synonyms in the XQuery grammar before you close XQuery 3.1 !?
Otherwise we’ll hear for another 100 years that XQuery has nothing to do with SQL while THIS language
describe in this paper DOES. (sic!)
It’s dumb, but that’s how it is.
Pretty please !???
Thanks
Dana

Post by daniela florescu
Ihe,
you asked why XQuery is not more popular.
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like <https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like>
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it does, and what is good at.
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness, productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.

_______________________________________________
http://x-query.com/mailman/listinfo/talk <http://x-query.com/mailman/listinfo/talk>

[daniela florescu]

Post by daniela florescu
1. the principles of XQuery are orthogonal to XML itself and

My colleague and friend Donald Kossmann used to tell his students :
“there are two wrong things with the name XQuery: the “X” and the “Query”.

The “X” because it’s not only for XML, it’s for semi-structured data in general, and
the “Query” because it’s not only a query language, it’s a general data processing language.

:-)

Best
Dana
_______________________________________________
***@x-query.com
http://x-query.c

[Pavel Minaev]

I don't mind at all.

And of course XQuery (well, the parts that are not tied to XDM) is
orthogonal to XML. The way I see it, FLWOR syntax (as well as LINQ etc) is
basically just a very fancy sequence comprehension, and obviously sequences
and trees aren't XML-specific at all.

FWIW, back when I first saw XQuery (which was after LINQ for me), I
remember being somewhat surprised at the choice of "return" especially for
a keyword. It's not that it was unclear what it did, and the mapping to the
more familiar syntax was obvious. It's just that it implies some transfer
of control traditionally, which is obviously not the case here. "yield", as
used in Scala and F#, made a lot more sense.

I don't mind "for" as much, on the other hand, seeing how it's pretty
commonly used for sequence comprehensions elsewhere (Python, Scala, F#
...). I think the main reason why LINQ used "from" instead is to induce a
sense of familiarity for those coming from SQL background, which was 99% of
its target audience.

Side note: I don't think that standalone query languages make much sense
now that integrated syntax for sequence comprehensions has become
mainstream for high-level languages. It's just too much effort to do
explicit interop from, say, Java, C# and JS, to XQuery or JSONiq, and it
introduces a new syntax for concepts which are already represented in the
host language, with said syntax being constrained to one narrow domain
(XML/JSON). I think that the way forward is going to be more like LINQ -
taking (hopefully!) the lessons learned from dedicated query languages and
seamlessly integrating them into the language such that they can work on
its native data model, and mapping XML, JSON etc to that model as
transparently as possible.

Post by daniela florescu
Pavel, I hope you don’t mind that I forward your personal email to the main list.
Thanks for the links.
Yes, I am of course aware of the history of it, but I don’t think everyone else is.
(I even gave feedback to Erik Meijer before he submitted this proposal to
Bill Gates
. and I lost
a beer with him because we bet who among the two of us knows XML Schema
better — and he did :-).
1. the principles of XQuery are orthogonal to XML itself and
2. spelling FROM and SELECT seems to matter to people A LOT.
Best regards
Dana
https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx
Which, ironically, came out of an earlier Microsoft project called Cω (and
specifically, the subset of it called X#), which was basically an attempt
https://msdn.microsoft.com/en-us/library/ms974195.aspx
It became LINQ after it dropped ties to XDM and was generalized to operate
on arbitrary CLI object graphs instead.

Post by daniela florescu
http://queue.acm.org/detail.cfm?id=1961297
Just look at the examples.
Does everyone see what I see !? :-)
Those are just FLWOR expressions, with FOR spelled as FROM and RETURN spelled
as SELECT.
Can we PLEASE add those as synonyms in the XQuery grammar before you close XQuery 3.1 !?
Otherwise we’ll hear for another 100 years that XQuery has nothing to do
with SQL while THIS language
describe in this paper DOES. (sic!)
It’s dumb, but that’s how it is.
Pretty please !???
Thanks
Dana
Ihe,
you asked why XQuery is not more popular.
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some
meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it
does, and what is good at.
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at
MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness,
productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[daniela florescu]

Pavel,

NOW we are talking about interesting research !!!! :-))))

Sequence comprehension. Here we go
.:-)

Yes, sequence comprehension (it’s called monoid comprehension in
theoretical computer science
.) IS the one of the most important things
for a high level/declarative language.

However, several important remarks:

============

1. Linq is NOT a new idea. My PhD thesis was (partially) about adding SELECT-FROM-WHERE
as an expression to C++. I did it as a pre-processor at that time — remember, it was 1994.
Others did it too, decades before me !!!! Remember Pascal-R !??? http://archive.nyu.edu/fda/handle/2451/14595 <http://archive.nyu.edu/fda/handle/2451/14595>
(probably not, because you are too young..)

2. in 2015 XQuery has FROM FAR (!!!!!!!!!!) the most advanced, expressive-wise, AND
the cleanest semantic-wise version of sequence comprehension.

Every other language should use THAT instead of half-baking Yet-Another-Sequence-Comprehension.

The FLWOR of XQuery is semantically very clean, has nothing to do with XML, is compositional and elegant,
and, moreover, optimizable.


3. The discussion : should we add sequence comprehension to an exiting imperative language vs.
adding scripting extensions to XQuery/high level functional language with sequence comprehension
is again old as the times.

see for example:
http://dl.acm.org/citation.cfm?id=1142597&dl=ACM&coll=DL&CFID=522088426&CFTOKEN=80093091 <http://dl.acm.org/citation.cfm?id=1142597&dl=ACM&coll=DL&CFID=522088426&CFTOKEN=80093091>
( Donald Kossmann and I had a detailed tutorial at Sigmod 2006, and the slides were REALLY funny, if I remember correctly)

This was in 2006. (when you read the paper just do a global replace “XML” with “semi-structured data”)

Now, in 2015, I think BOTH alternatives are good, and viable.

For example, adding a FLWOR expression to Javascript is a worthwile thing to do, and I am implementing this as we speak.

Another example: what do you think that Oracle Fusion was (multi-billion failure..)!? Just a rewrite of Oracle application, where the code instead of being written
in PL-SQL was written in Java with SQL.

Same story/question again.

4. Sequence comprehension is necessary for processing data in general, structured or non-structured.

However, for processing ****SEMI-STRUCTURED DATA**** you need EXTRA other features, non-existing in
ANY other language except XQuery and XSLT.

I tried to put a list of features needed for semi-structured data in a previous email to this list, 
.
http://x-query.com/pipermail/talk/2015-May/004718.html
http://x-query.com/pipermail/talk/2015-May/004719.html


..but of course I did it fast, and forgot some of the necessary features.
In particular, I forgot the ability to do metadata search (aka path expressions where you don’t know the data structure).

===========



So, yep. Lots of Computer Science history.


This is why it is DAMN SAD to see those barbarians starting to query JSON in 2015 like it’s 1995 again 

.



Best
Dana

Post by Pavel Minaev
I don't mind at all.
And of course XQuery (well, the parts that are not tied to XDM) is orthogonal to XML. The way I see it, FLWOR syntax (as well as LINQ etc) is basically just a very fancy sequence comprehension, and obviously sequences and trees aren't XML-specific at all.
FWIW, back when I first saw XQuery (which was after LINQ for me), I remember being somewhat surprised at the choice of "return" especially for a keyword. It's not that it was unclear what it did, and the mapping to the more familiar syntax was obvious. It's just that it implies some transfer of control traditionally, which is obviously not the case here. "yield", as used in Scala and F#, made a lot more sense.
I don't mind "for" as much, on the other hand, seeing how it's pretty commonly used for sequence comprehensions elsewhere (Python, Scala, F# ...). I think the main reason why LINQ used "from" instead is to induce a sense of familiarity for those coming from SQL background, which was 99% of its target audience.
Side note: I don't think that standalone query languages make much sense now that integrated syntax for sequence comprehensions has become mainstream for high-level languages. It's just too much effort to do explicit interop from, say, Java, C# and JS, to XQuery or JSONiq, and it introduces a new syntax for concepts which are already represented in the host language, with said syntax being constrained to one narrow domain (XML/JSON). I think that the way forward is going to be more like LINQ - taking (hopefully!) the lessons learned from dedicated query languages and seamlessly integrating them into the language such that they can work on its native data model, and mapping XML, JSON etc to that model as transparently as possible.
Pavel, I hope you don’t mind that I forward your personal email to the main list.
Thanks for the links.
Yes, I am of course aware of the history of it, but I don’t think everyone else is.
(I even gave feedback to Erik Meijer before he submitted this proposal to Bill Gates
. and I lost
a beer with him because we bet who among the two of us knows XML Schema better — and he did :-).
1. the principles of XQuery are orthogonal to XML itself and
2. spelling FROM and SELECT seems to matter to people A LOT.
Best regards
Dana

https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx <https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx>
https://msdn.microsoft.com/en-us/library/ms974195.aspx <https://msdn.microsoft.com/en-us/library/ms974195.aspx>
It became LINQ after it dropped ties to XDM and was generalized to operate on arbitrary CLI object graphs instead.
http://queue.acm.org/detail.cfm?id=1961297 <http://queue.acm.org/detail.cfm?id=1961297>
Just look at the examples.
Does everyone see what I see !? :-)
Those are just FLWOR expressions, with FOR spelled as FROM and RETURN spelled
as SELECT.
Can we PLEASE add those as synonyms in the XQuery grammar before you close XQuery 3.1 !?
Otherwise we’ll hear for another 100 years that XQuery has nothing to do with SQL while THIS language
describe in this paper DOES. (sic!)
It’s dumb, but that’s how it is.
Pretty please !???
Thanks
Dana

Post by daniela florescu
Ihe,
you asked why XQuery is not more popular.
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like <https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like>
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it does, and what is good at.
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness, productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.

_______________________________________________
http://x-query.com/mailman/listinfo/talk <http://x-query.com/mailman/listinfo/talk>

[Pavel Minaev]

I think that was distinguishes LINQ is not that it was the first such
experiment, but that it was the first one that happened in a mainstream
non-declarative language (as opposed to a fork of one) *and *was widely
adopted by the community of that language.

As far as what's "better", I think that there are really two different
questions here: what's "better" in The Right Way sense, and what's "better"
in a "how do you get people to use this" sense. For the latter, there's no
doubt in my mind that the way there is to get existing mainstream languages
(which are all imperative at heart) to adopt more declarative goodness,
including comprehensions/queries, as a subset - it's a much gentler
transition for existing users of those languages, and more importantly,
it's like a new instrument magically appearing in a toolbox that they
already use daily - at some point they're bound to pick it up and use it
even just out of sheer curiosity. It's much harder to get someone
interested in a completely new language in comparison.

I'm curious as to what exactly you believe to be features that are
necessary to semi-structured data that are unique to XQuery. I've looked at
the posts that you've linked to, and e.g. conditionals/typeswitches,
functions and recursive functions, and try/catch, are there in pretty much
any imperative language with sequence comprehensions, and can usually be
combined more or less seamlessly (though if I had to bring up a
particularly good example of all of these, it would be F# computation
expressions, which are also nicely generalized and extensible - see
https://msdn.microsoft.com/en-us/library/dd233209.aspx and
https://msdn.microsoft.com/en-us/library/dd233182.aspx). Implicit type
conversions are the only point I can think of that's truly unique here, but
I think that to what degree that is appropriate is highly subjective - it's
just as easy to get messed-up output with no clear indication of what the
problem was due to an unfortunate conversion producing garbage somewhere in
the middle of a particularly complicated query. It all depends on the data.

Post by daniela florescu
Pavel,
NOW we are talking about interesting research !!!! :-))))
Sequence comprehension. Here we go
.:-)
Yes, sequence comprehension (it’s called monoid comprehension in
theoretical computer science
.) IS the one of the most important things
for a high level/declarative language.
============
1. Linq is NOT a new idea. My PhD thesis was (partially) about adding SELECT-FROM-WHERE
as an expression to C++. I did it as a pre-processor at that time —
remember, it was 1994.
Others did it too, decades before me !!!! Remember Pascal-R !???
http://archive.nyu.edu/fda/handle/2451/14595
(probably not, because you are too young..)
2. in 2015 XQuery has FROM FAR (!!!!!!!!!!) the most advanced,
expressive-wise, AND
the cleanest semantic-wise version of sequence comprehension.
Every other language should use THAT instead of half-baking
Yet-Another-Sequence-Comprehension.
The FLWOR of XQuery is semantically very clean, has nothing to do with
XML, is compositional and elegant,
and, moreover, optimizable.
3. The discussion : should we add sequence comprehension to an exiting
imperative language vs.
adding scripting extensions to XQuery/high level functional language with
sequence comprehension
is again old as the times.
http://dl.acm.org/citation.cfm?id=1142597&dl=ACM&coll=DL&CFID=522088426&CFTOKEN=80093091
( Donald Kossmann and I had a detailed tutorial at Sigmod 2006, and the
slides were REALLY funny, if I remember correctly)
This was in 2006. (when you read the paper just do a global replace “XML”
with “semi-structured data”)
Now, in 2015, I think BOTH alternatives are good, and viable.
For example, adding a FLWOR expression to Javascript is a worthwile thing
to do, and I am implementing this as we speak.
Another example: what do you think that Oracle Fusion was (multi-billion
failure..)!? Just a rewrite of Oracle application, where the code instead
of being written
in PL-SQL was written in Java with SQL.
Same story/question again.
4. Sequence comprehension is necessary for processing data in general,
structured or non-structured.
However, for processing ****SEMI-STRUCTURED DATA**** you need EXTRA other
features, non-existing in
ANY other language except XQuery and XSLT.
I tried to put a list of features needed for semi-structured data in a
previous email to this list, 
.
http://x-query.com/pipermail/talk/2015-May/004718.html
http://x-query.com/pipermail/talk/2015-May/004719.html

..but of course I did it fast, and forgot some of the necessary features.
In particular, I forgot the ability to do metadata search (aka path
expressions where you don’t know the data structure).
===========
So, yep. Lots of Computer Science history.
This is why it is DAMN SAD to see those barbarians starting to query JSON
in 2015 like it’s 1995 again 

.
Best
Dana
I don't mind at all.
And of course XQuery (well, the parts that are not tied to XDM) is
orthogonal to XML. The way I see it, FLWOR syntax (as well as LINQ etc) is
basically just a very fancy sequence comprehension, and obviously sequences
and trees aren't XML-specific at all.
FWIW, back when I first saw XQuery (which was after LINQ for me), I
remember being somewhat surprised at the choice of "return" especially for
a keyword. It's not that it was unclear what it did, and the mapping to the
more familiar syntax was obvious. It's just that it implies some transfer
of control traditionally, which is obviously not the case here. "yield", as
used in Scala and F#, made a lot more sense.
I don't mind "for" as much, on the other hand, seeing how it's pretty
commonly used for sequence comprehensions elsewhere (Python, Scala, F#
...). I think the main reason why LINQ used "from" instead is to induce a
sense of familiarity for those coming from SQL background, which was 99% of
its target audience.
Side note: I don't think that standalone query languages make much sense
now that integrated syntax for sequence comprehensions has become
mainstream for high-level languages. It's just too much effort to do
explicit interop from, say, Java, C# and JS, to XQuery or JSONiq, and it
introduces a new syntax for concepts which are already represented in the
host language, with said syntax being constrained to one narrow domain
(XML/JSON). I think that the way forward is going to be more like LINQ -
taking (hopefully!) the lessons learned from dedicated query languages and
seamlessly integrating them into the language such that they can work on
its native data model, and mapping XML, JSON etc to that model as
transparently as possible.

Post by daniela florescu
Pavel, I hope you don’t mind that I forward your personal email to the main list.
Thanks for the links.
Yes, I am of course aware of the history of it, but I don’t think everyone else is.
(I even gave feedback to Erik Meijer before he submitted this proposal to
Bill Gates
. and I lost
a beer with him because we bet who among the two of us knows XML Schema
better — and he did :-).
1. the principles of XQuery are orthogonal to XML itself and
2. spelling FROM and SELECT seems to matter to people A LOT.
Best regards
Dana
https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx
Which, ironically, came out of an earlier Microsoft project called Cω
(and specifically, the subset of it called X#), which was basically an
https://msdn.microsoft.com/en-us/library/ms974195.aspx
It became LINQ after it dropped ties to XDM and was generalized to
operate on arbitrary CLI object graphs instead.

Post by daniela florescu
One thing that did strike me is the link cited in one of the comments
http://queue.acm.org/detail.cfm?id=1961297
Just look at the examples.
Does everyone see what I see !? :-)
Those are just FLWOR expressions, with FOR spelled as FROM and RETURN spelled
as SELECT.
Can we PLEASE add those as synonyms in the XQuery grammar before you
close XQuery 3.1 !?
Otherwise we’ll hear for another 100 years that XQuery has nothing to do
with SQL while THIS language
describe in this paper DOES. (sic!)
It’s dumb, but that’s how it is.
Pretty please !???
Thanks
Dana
Ihe,
you asked why XQuery is not more popular.
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some
meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it
does, and what is good at.
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at
MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness,
productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.
_______________________________________________
http://x-query.com/mailman/listinfo/talk

_______________________________________________
http://x-query.com/mailman/listinfo/talk

[daniela florescu]

Pavel,

I’ll respond to this in two different emails, as it contains two different important questions.

Best
Dana

I think that was distinguishes LINQ is not that it was the first such experiment, but that it was the first one that happened in a mainstream non-declarative language (as opposed to a fork of one) and was widely adopted by the community of that language.
As far as what's "better", I think that there are really two different questions here: what's "better" in The Right Way sense, and what's "better" in a "how do you get people to use this" sense. For the latter, there's no doubt in my mind that the way there is to get existing mainstream languages (which are all imperative at heart) to adopt more declarative goodness, including comprehensions/queries, as a subset - it's a much gentler transition for existing users of those languages, and more importantly, it's like a new instrument magically appearing in a toolbox that they already use daily - at some point they're bound to pick it up and use it even just out of sheer curiosity. It's much harder to get someone interested in a completely new language in comparison.
I'm curious as to what exactly you believe to be features that are necessary to semi-structured data that are unique to XQuery. I've looked at the posts that you've linked to, and e.g. conditionals/typeswitches, functions and recursive functions, and try/catch, are there in pretty much any imperative language with sequence comprehensions, and can usually be combined more or less seamlessly (though if I had to bring up a particularly good example of all of these, it would be F# computation expressions, which are also nicely generalized and extensible - see https://msdn.microsoft.com/en-us/library/dd233209.aspx <https://msdn.microsoft.com/en-us/library/dd233209.aspx> and https://msdn.microsoft.com/en-us/library/dd233182.aspx <https://msdn.microsoft.com/en-us/library/dd233182.aspx>). Implicit type conversions are the only point I can think of that's truly unique here, but I think that to what degree that is appropriate is highly subjective - it's just as easy to get messed-up output with no clear indication of what the problem was due to an unfortunate conversion producing garbage somewhere in the middle of a particularly complicated query. It all depends on the data.
Pavel,
NOW we are talking about interesting research !!!! :-))))
Sequence comprehension. Here we go
.:-)
Yes, sequence comprehension (it’s called monoid comprehension in
theoretical computer science
.) IS the one of the most important things
for a high level/declarative language.
============
1. Linq is NOT a new idea. My PhD thesis was (partially) about adding SELECT-FROM-WHERE
as an expression to C++. I did it as a pre-processor at that time — remember, it was 1994.
Others did it too, decades before me !!!! Remember Pascal-R !??? http://archive.nyu.edu/fda/handle/2451/14595 <http://archive.nyu.edu/fda/handle/2451/14595>
(probably not, because you are too young..)
2. in 2015 XQuery has FROM FAR (!!!!!!!!!!) the most advanced, expressive-wise, AND
the cleanest semantic-wise version of sequence comprehension.
Every other language should use THAT instead of half-baking Yet-Another-Sequence-Comprehension.
The FLWOR of XQuery is semantically very clean, has nothing to do with XML, is compositional and elegant,
and, moreover, optimizable.
3. The discussion : should we add sequence comprehension to an exiting imperative language vs.
adding scripting extensions to XQuery/high level functional language with sequence comprehension
is again old as the times.
http://dl.acm.org/citation.cfm?id=1142597&dl=ACM&coll=DL&CFID=522088426&CFTOKEN=80093091 <http://dl.acm.org/citation.cfm?id=1142597&dl=ACM&coll=DL&CFID=522088426&CFTOKEN=80093091>
( Donald Kossmann and I had a detailed tutorial at Sigmod 2006, and the slides were REALLY funny, if I remember correctly)
This was in 2006. (when you read the paper just do a global replace “XML” with “semi-structured data”)
Now, in 2015, I think BOTH alternatives are good, and viable.
For example, adding a FLWOR expression to Javascript is a worthwile thing to do, and I am implementing this as we speak.
Another example: what do you think that Oracle Fusion was (multi-billion failure..)!? Just a rewrite of Oracle application, where the code instead of being written
in PL-SQL was written in Java with SQL.
Same story/question again.
4. Sequence comprehension is necessary for processing data in general, structured or non-structured.
However, for processing ****SEMI-STRUCTURED DATA**** you need EXTRA other features, non-existing in
ANY other language except XQuery and XSLT.
I tried to put a list of features needed for semi-structured data in a previous email to this list, 
.
http://x-query.com/pipermail/talk/2015-May/004718.html <http://x-query.com/pipermail/talk/2015-May/004718.html>
http://x-query.com/pipermail/talk/2015-May/004719.html <http://x-query.com/pipermail/talk/2015-May/004719.html>

..but of course I did it fast, and forgot some of the necessary features.
In particular, I forgot the ability to do metadata search (aka path expressions where you don’t know the data structure).
===========
So, yep. Lots of Computer Science history.
This is why it is DAMN SAD to see those barbarians starting to query JSON in 2015 like it’s 1995 again 

.
Best
Dana

Post by Pavel Minaev
I don't mind at all.
And of course XQuery (well, the parts that are not tied to XDM) is orthogonal to XML. The way I see it, FLWOR syntax (as well as LINQ etc) is basically just a very fancy sequence comprehension, and obviously sequences and trees aren't XML-specific at all.
FWIW, back when I first saw XQuery (which was after LINQ for me), I remember being somewhat surprised at the choice of "return" especially for a keyword. It's not that it was unclear what it did, and the mapping to the more familiar syntax was obvious. It's just that it implies some transfer of control traditionally, which is obviously not the case here. "yield", as used in Scala and F#, made a lot more sense.
I don't mind "for" as much, on the other hand, seeing how it's pretty commonly used for sequence comprehensions elsewhere (Python, Scala, F# ...). I think the main reason why LINQ used "from" instead is to induce a sense of familiarity for those coming from SQL background, which was 99% of its target audience.
Side note: I don't think that standalone query languages make much sense now that integrated syntax for sequence comprehensions has become mainstream for high-level languages. It's just too much effort to do explicit interop from, say, Java, C# and JS, to XQuery or JSONiq, and it introduces a new syntax for concepts which are already represented in the host language, with said syntax being constrained to one narrow domain (XML/JSON). I think that the way forward is going to be more like LINQ - taking (hopefully!) the lessons learned from dedicated query languages and seamlessly integrating them into the language such that they can work on its native data model, and mapping XML, JSON etc to that model as transparently as possible.
Pavel, I hope you don’t mind that I forward your personal email to the main list.
Thanks for the links.
Yes, I am of course aware of the history of it, but I don’t think everyone else is.
(I even gave feedback to Erik Meijer before he submitted this proposal to Bill Gates
. and I lost
a beer with him because we bet who among the two of us knows XML Schema better — and he did :-).
1. the principles of XQuery are orthogonal to XML itself and
2. spelling FROM and SELECT seems to matter to people A LOT.
Best regards
Dana

https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx <https://msdn.microsoft.com/en-us/library/bb397896(v=vs.140).aspx>
https://msdn.microsoft.com/en-us/library/ms974195.aspx <https://msdn.microsoft.com/en-us/library/ms974195.aspx>
It became LINQ after it dropped ties to XDM and was generalized to operate on arbitrary CLI object graphs instead.
http://queue.acm.org/detail.cfm?id=1961297 <http://queue.acm.org/detail.cfm?id=1961297>
Just look at the examples.
Does everyone see what I see !? :-)
Those are just FLWOR expressions, with FOR spelled as FROM and RETURN spelled
as SELECT.
Can we PLEASE add those as synonyms in the XQuery grammar before you close XQuery 3.1 !?
Otherwise we’ll hear for another 100 years that XQuery has nothing to do with SQL while THIS language
describe in this paper DOES. (sic!)
It’s dumb, but that’s how it is.
Pretty please !???
Thanks
Dana

Post by daniela florescu
Ihe,
you asked why XQuery is not more popular.
Here is another striking answer to your question: NOBODY KNOWS IT EXISTS.
Just look at this example.
https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like <https://www.linkedin.com/pulse/sql-vs-discrepancy-somil-asthana?trk=hp-feed-article-title-like>
I quote: "Indisputably, there may be people who are working on Non Relational
Algebra and Non Tuple Relational Calculus, its just that we do not know them.”
[[[Can someone just answer this guy, so I don’t have to insult him/her !?
Because I feel a really strong urge
.I’ll try to breathe and do some meditation
.]]]
In fact, I know that this is not his/her limitation.
It’s our OWN failure to explain to the world what XQuery is, what it does, and what is good at.
Best regards
Dana
P.S. And after that, please DON’T ask me why I am SO pissed off at MarkLogic who pretend they never ate the garlic, not does
their mouth smell of garlic
..
They MADE all their money out of the power of XQuery (expressiveness, productivity, etc), yet they pretend they’ve never heard of it
.
That’s something that REALLY gets me angry.
And this will come back to bait them on the business side very badly too.
Oracle would have NEVER done the same thing about SQL
..just saying.

_______________________________________________
http://x-query.com/mailman/listinfo/talk <http://x-query.com/mailman/listinfo/talk>

_______________________________________________
http://x-query.com/mailman/listinfo/talk <http://x-query.com/mailman/listinfo/talk>

[daniela florescu]

As far as what's "better", I think that there are really two different questions here: what's "better" in The Right Way sense, and what's "better" in a "how do you get people to use this" sense. For the latter, there's no doubt in my mind that the way there is to get existing mainstream languages (which are all imperative at heart) to adopt more declarative goodness, including comprehensions/queries, as a subset - it's a much gentler transition for existing users of those languages, and more importantly, it's like a new instrument magically appearing in a toolbox that they already use daily - at some point they're bound to pick it up and use it even just out of sheer curiosity. It's much harder to get someone interested in a completely new language in comparison.

As I said in my previous email.

Now in 2015 I realize that developers have their “favorite” language, and making them change it’s close to
impossible, so the optimal is to give them better tools to do their job in THE language they are more productive in.

So if this is C#, or Javascript, or whatever…. let it be.
(ever crossed your mind why in an EC commerce site when you sort on price/sometinh, it GOES BACK TO THE SERVER most of the time to
do a simple sort on 20 products !???? Sounds stupid no, ? well, that’s because Javascript doesn’t HAVE a sortby ….so …)

It’s better for those imperative languages to HAVE a sequence comprehension then NOT to have it. It makes the developers
in those languages (who otherwise wouldn’t have changed anyway…) more productive….

Now, the fact that 90% of developers like comfort and/or are learning impaired and cannot learn something new
does NOT mean that ALL developers are lazy and/or learning impaired and cannot learn something new…..

For those who CAN learn something new, or for the fresh new students with blank, fresh new minds…..well, teach them something better
and more productive.

And that would be (according to me..) the OTHER way around: a language 95% declarative + 5% imperative.

As I gave you as an example: when Oracle Fusion started the idea was that Java, being easier to program in then the “declarative” PL-SQL…. the
applications will be MUCH better.

Well, 10 years later, multi-billion investment with epsilon returns, it turns out that the PL-SQL version was MUCH better then the Java version.
(1000 times more code, more bugs, less productive, more developers, more politics and decision makers, etc, etc — as simple as that ..)

A failure that Oracle never talks about ….. just Oracle Fusion silently disappeared from the marketing slides :-)

=====

So, no, I would not take either side at this point.

Best
Dana








_______________________________________________
***@x-query.com
http:/

[Pavel Minaev]

It's not really a matter of learning something new - it's more about
practicality. I know XQuery, Haskell etc, for example, but for purely
pragmatic reasons (such as the need to use existing libraries, and the
desire to use them with ease - i.e. without mucking around with FFI, and
the associated impedance mismatch that using a library written in an
imperative language like C from a pure functional language causes), and
because I'm not working on that code alone and other people should be able
to pick it up where I left it and continue working, 99% of the time I'm
writing in C++, C#, Python etc.

Simply put, the requirement to work with other people and to interoperate
with other systems makes more conventional stuff *more productive*,
ironically, while unconventional things that break away from the mold are
only more productive in a hypothetical universe where they are all by
themselves and everyone and everything is starting from a clean slate
(which does in fact occasionally happen IRL when there appears a new niche
that is not [yet] tightly connected to everything else).

That's why I love the fact that functional and declarative stuff is
creeping into those mainstream imperative languages. It gradually gets me
to the point where I can write neat declarative code *in production*, and
no-one objects because they don't realize that it's radically different
from what they're used to! (and those who do, are generally in the same
boat as me and enjoy it the same). For them, it's just some new syntax that
got added to the language they know and use, and so obviously it's okay by
default.

I think we'll eventually get to your 95%+5% end goal this way anyway. It
will just be slow enough that nobody will notice other than in retrospect.

Post by Pavel Minaev

Post by Pavel Minaev
As far as what's "better", I think that there are really two different

questions here: what's "better" in The Right Way sense, and what's "better"
in a "how do you get people to use this" sense. For the latter, there's no
doubt in my mind that the way there is to get existing mainstream languages
(which are all imperative at heart) to adopt more declarative goodness,
including comprehensions/queries, as a subset - it's a much gentler
transition for existing users of those languages, and more importantly,
it's like a new instrument magically appearing in a toolbox that they
already use daily - at some point they're bound to pick it up and use it
even just out of sheer curiosity. It's much harder to get someone
interested in a completely new language in comparison.
As I said in my previous email.
Now in 2015 I realize that developers have their “favorite” language, and
making them change it’s close to
impossible, so the optimal is to give them better tools to do their job in
THE language they are more productive in.
So if this is C#, or Javascript, or whatever
. let it be.
(ever crossed your mind why in an EC commerce site when you sort on
price/sometinh, it GOES BACK TO THE SERVER most of the time to
do a simple sort on 20 products !???? Sounds stupid no, ? well, that’s
because Javascript doesn’t HAVE a sortby 
.so 
)
It’s better for those imperative languages to HAVE a sequence
comprehension then NOT to have it. It makes the developers
in those languages (who otherwise wouldn’t have changed anyway
) more
productive
.
Now, the fact that 90% of developers like comfort and/or are learning
impaired and cannot learn something new
does NOT mean that ALL developers are lazy and/or learning impaired and
cannot learn something new
..
For those who CAN learn something new, or for the fresh new students with
blank, fresh new minds
..well, teach them something better
and more productive.
And that would be (according to me..) the OTHER way around: a language 95%
declarative + 5% imperative.
As I gave you as an example: when Oracle Fusion started the idea was that
Java, being easier to program in then the “declarative” PL-SQL
. the
applications will be MUCH better.
Well, 10 years later, multi-billion investment with epsilon returns, it
turns out that the PL-SQL version was MUCH better then the Java version.
(1000 times more code, more bugs, less productive, more developers, more
politics and decision makers, etc, etc — as simple as that ..)
A failure that Oracle never talks about 
.. just Oracle Fusion silently
disappeared from the marketing slides :-)
=====
So, no, I would not take either side at this point.
Best
Dana

[daniela florescu]

I think we'll eventually get to your 95%+5% end goal this way anyway. It will just be slow enough that nobody will notice other than in retrospect.

If it goes slowly, so be it. But I learned that CS goes through growth spurts, like children.

You never know when the next “wave” comes, or why. (e.g. it’s very likely the relational databases wouldn’t have ever seen the
light at all if they wouldn’t have had an egomaniac and out-of-range greedy individual like Larry Ellison pushing for them
)

In fact, there is SOME correlation that I can observe: in period of “financial bubble” there is almost ZERO new technology attempted.

People are too busy listening to the marketing noise to pay attention to the algorithms behind the noise 
.

People pay too much attention to the gazzilions of bizzilions of dollars
to pay attention to functional vs. imperative
and the impact of monads :-)

It’s only crazy people like us
...

Best
Dana

As far as what's "better", I think that there are really two different questions here: what's "better" in The Right Way sense, and what's "better" in a "how do you get people to use this" sense. For the latter, there's no doubt in my mind that the way there is to get existing mainstream languages (which are all imperative at heart) to adopt more declarative goodness, including comprehensions/queries, as a subset - it's a much gentler transition for existing users of those languages, and more importantly, it's like a new instrument magically appearing in a toolbox that they already use daily - at some point they're bound to pick it up and use it even just out of sheer curiosity. It's much harder to get someone interested in a completely new language in comparison.

As I said in my previous email.
Now in 2015 I realize that developers have their “favorite” language, and making them change it’s close to
impossible, so the optimal is to give them better tools to do their job in THE language they are more productive in.
So if this is C#, or Javascript, or whatever
. let it be.
(ever crossed your mind why in an EC commerce site when you sort on price/sometinh, it GOES BACK TO THE SERVER most of the time to
do a simple sort on 20 products !???? Sounds stupid no, ? well, that’s because Javascript doesn’t HAVE a sortby 
.so 
)
It’s better for those imperative languages to HAVE a sequence comprehension then NOT to have it. It makes the developers
in those languages (who otherwise wouldn’t have changed anyway
) more productive
.
Now, the fact that 90% of developers like comfort and/or are learning impaired and cannot learn something new
does NOT mean that ALL developers are lazy and/or learning impaired and cannot learn something new
..
For those who CAN learn something new, or for the fresh new students with blank, fresh new minds
..well, teach them something better
and more productive.
And that would be (according to me..) the OTHER way around: a language 95% declarative + 5% imperative.
As I gave you as an example: when Oracle Fusion started the idea was that Java, being easier to program in then the “declarative” PL-SQL
. the
applications will be MUCH better.
Well, 10 years later, multi-billion investment with epsilon returns, it turns out that the PL-SQL version was MUCH better then the Java version.
(1000 times more code, more bugs, less productive, more developers, more politics and decision makers, etc, etc — as simple as that ..)
A failure that Oracle never talks about 
.. just Oracle Fusion silently disappeared from the marketing slides :-)
=====
So, no, I would not take either side at this point.
Best
Dana
_______________________________________________
http://x-query.com/mailman/listinfo/talk

[Ihe Onwuka]

Post by Pavel Minaev
That's why I love the fact that functional and declarative stuff is
creeping into those mainstream imperative languages. It gradually gets me
to the point where I can write neat declarative code *in production*, and
no-one objects because they don't realize that it's radically different
from what they're used to! (and those who do, are generally in the same
boat as me and enjoy it the same). For them, it's just some new syntax that
got added to the language they know and use, and so obviously it's okay by
default.

I don't think this is generally applicable. For example look at Google's
Python standards

https://google-styleguide.googlecode.com/svn/trunk/pyguide.html

Comprehensions are only allowed for "simple cases" otherwise use a loop. No
mention of fold anywhere in the document which I interpret as meaning don't
use it.

[Pavel Minaev]

Google is ultra-conservative in their approach to coding. It's not just the
FP stuff, their C++ coding standard is equally ridiculous for other
reasons. I suspect it's also why they use Java so much, and I expect that
they will ban lambdas there when they move on to Java 8. So I am inclined
to treat them as a more extreme case than normal. Perhaps they do have good
reasons to do things that way, but even if so, I don't think those reasons
apply to most.

For comparison, in my experience working at Microsoft, declarative
operations on sequences (either comprehensions, or just direct use of
map/filter/... or local equivalents thereof) are widely accepted and used
in production C# and Python code, and have been for at least six years. On
my team, for example, the use of an explicit loop where a comprehension
would suffice would generally be flagged in a code review right away. The
only objections that I've ever heard come from perf hawks, and that is
sometimes justifiable, but is really an implementation deficiency - there's
no reason why a compiler cannot optimize a comprehension to be as efficient
as a straight loop in most cases, and generally be better than it currently
is in all cases; it just wasn't a goal for the C# compiler guys (and of
course it's not a concern for Python at all).

Speaking of "hard to understand" and "not as performant" arguments popping
up... I remember that being quite frequent in the community back when all
this stuff was introduced into the language and the framework. Three years
later, it was already a distinctive, if still loud, minority. Now, you
still hear it occasionally, but most people just shrug at it, and lambdas
and LINQ are just part of idiomatic C#. Same thing happened with type
inference ("var") in C#, and lambdas in C++, and many other things. And I
think we are seeing a replay with lambdas in Java 8 (it's really funny to
read the disapproving blog posts about Java lambdas today; you could
s/Java/C#/ and they'd be word for word the same as what I've read back in
2008).

Thing is, programmers are a pragmatic bunch. If it really does work better
(= lets them be more productive; you don't need to be a scientist to gauge
that), they'll get used to it, and will eventually grow fond of it, so long
as you don't force them to go all in on it right away. A certain amount of
bickering is to be expected, and there will always be stalwart holdouts (I
mean, we still have people signing a petition to bring back VB6, 17 years
later!), but overall the industry moves on. I'd rather be an optimist on
account of the direction of that movement and its ultimate destination,
than a pessimist on account of its speed, or the occasional detour along
the way.

And it's not like all this is recent, either. Weren't there very similar
objections against structured programming back in the day? You know, why
complicate language with these new-fangled and hard to understand loops if
a simple conditional jump is just as expressive and can be hand-optimized
much better than even the fanciest compiler can manage? And yet, here we
are.

Post by Ihe Onwuka

Post by Pavel Minaev
That's why I love the fact that functional and declarative stuff is
creeping into those mainstream imperative languages. It gradually gets me
to the point where I can write neat declarative code *in production*,
and no-one objects because they don't realize that it's radically different
from what they're used to! (and those who do, are generally in the same
boat as me and enjoy it the same). For them, it's just some new syntax that
got added to the language they know and use, and so obviously it's okay by
default.

I don't think this is generally applicable. For example look at Google's
Python standards
https://google-styleguide.googlecode.com/svn/trunk/pyguide.html
Comprehensions are only allowed for "simple cases" otherwise use a loop.
No mention of fold anywhere in the document which I interpret as meaning
don't use it.

[Ihe Onwuka]

Post by Pavel Minaev
Google is ultra-conservative in their approach to coding.

Google and every bank and insurance company under the sun and god knows how
many other places but certainly the majority.

<snip>

but not because it's irrelevant or that I don't agree ......see comments at
end

Post by Pavel Minaev
Thing is, programmers are a pragmatic bunch. If it really does work better
(= lets them be more productive; you don't need to be a scientist to gauge
that), they'll get used to it, and will eventually grow fond of it, so long
as you don't force them to go all in on it right away.

They are pragmatic in the sense that they generally won't bother learning
stuff if they don't have to and/or can't foresee being able to use it
because of the very reasons why you say you don't use XQuery/Haskell. At
some (i.e more than one) of the investment banks I've been, the standard
methodology for dealing with complexity is to crank out the debugger. When
a guy there tells you he is being pragmatic thats the sort of thing he
means.

Post by Pavel Minaev
A certain amount of bickering is to be expected, and there will always be
stalwart holdouts (I mean, we still have people signing a petition to bring
back VB6, 17 years later!), but overall the industry moves on. I'd rather
be an optimist on account of the direction of that movement and its
ultimate destination, than a pessimist on account of its speed, or the
occasional detour along the way.

Pavel if you work for Microsoft you are probably surrounded by smart
colleagues who have the capacity to get it which makes you an outlier
(sorry to use such a word). I remember sitting next to an MIT graduate on a
flight talking about the adjustment you have to make when you leave an
environment where you are surrounded by smart people so that you don't keep
saying DUHHHH! at the things and thoughts you encounter in the "real world".

Functional programming enforces a design methodology that has the benign
side effect facilitating the conquering of complexity BUT ONLY IF YOU GET
IT.
Those who don't will spend an entire weekend trying and failing to figure
out how to write a fibonacci function or come up with a 37 line solution if
asked to code Pascal's triangle.

In an imperative coding environment a programmer can usually fashion
something that works (or gives the outward impression that it works). This
is not the case with functional programming which entails a totally
different thought process. So whereas FP is more productive for those who
get it, it renders those who don't incapable of producing anything at all
and those who don't make up the majority of the programming populace.

It's always better to look and sound optimistic but the biggest influence
on what progress is made in the future will be how those in the present
think and in the main that has not really changed.

[Pavel Velikhov]

Ihe,

This is the most depressing post on programming I have read in years!

Best regards,
Pavel

Post by Pavel Minaev
Google is ultra-conservative in their approach to coding.
Google and every bank and insurance company under the sun and god knows how many other places but certainly the majority.
<snip>
but not because it's irrelevant or that I don't agree ......see comments at end
Thing is, programmers are a pragmatic bunch. If it really does work better (= lets them be more productive; you don't need to be a scientist to gauge that), they'll get used to it, and will eventually grow fond of it, so long as you don't force them to go all in on it right away.
They are pragmatic in the sense that they generally won't bother learning stuff if they don't have to and/or can't foresee being able to use it because of the very reasons why you say you don't use XQuery/Haskell. At some (i.e more than one) of the investment banks I've been, the standard methodology for dealing with complexity is to crank out the debugger. When a guy there tells you he is being pragmatic thats the sort of thing he means.
A certain amount of bickering is to be expected, and there will always be stalwart holdouts (I mean, we still have people signing a petition to bring back VB6, 17 years later!), but overall the industry moves on. I'd rather be an optimist on account of the direction of that movement and its ultimate destination, than a pessimist on account of its speed, or the occasional detour along the way.
Pavel if you work for Microsoft you are probably surrounded by smart colleagues who have the capacity to get it which makes you an outlier (sorry to use such a word). I remember sitting next to an MIT graduate on a flight talking about the adjustment you have to make when you leave an environment where you are surrounded by smart people so that you don't keep saying DUHHHH! at the things and thoughts you encounter in the "real world".
Functional programming enforces a design methodology that has the benign side effect facilitating the conquering of complexity BUT ONLY IF YOU GET IT.
Those who don't will spend an entire weekend trying and failing to figure out how to write a fibonacci function or come up with a 37 line solution if asked to code Pascal's triangle.
In an imperative coding environment a programmer can usually fashion something that works (or gives the outward impression that it works). This is not the case with functional programming which entails a totally different thought process. So whereas FP is more productive for those who get it, it renders those who don't incapable of producing anything at all and those who don't make up the majority of the programming populace.
It's always better to look and sound optimistic but the biggest influence on what progress is made in the future will be how those in the present think and in the main that has not really changed.
_______________________________________________
http://x-query.com/mailman/listinfo/talk

С уважеМОеЌ,
Павел ВелОхПв
***@gmail.com

[Ihe Onwuka]

Post by Pavel Minaev

Post by Pavel Minaev
As far as what's "better", I think that there are really two different

questions here: what's "better" in The Right Way sense, and what's "better"
in a "how do you get people to use this" sense. For the latter, there's no
doubt in my mind that the way there is to get existing mainstream languages
(which are all imperative at heart) to adopt more declarative goodness,
including comprehensions/queries, as a subset - it's a much gentler
transition for existing users of those languages, and more importantly,
it's like a new instrument magically appearing in a toolbox that they
already use daily - at some point they're bound to pick it up and use it
even just out of sheer curiosity. It's much harder to get someone
interested in a completely new language in comparison.
As I said in my previous email.
Now in 2015 I realize that developers have their “favorite” language, and
making them change it’s close to
impossible, so the optimal is to give them better tools to do their job in
THE language they are more productive in.
So if this is C#, or Javascript, or whatever
. let it be.
(ever crossed your mind why in an EC commerce site when you sort on
price/sometinh, it GOES BACK TO THE SERVER most of the time to
do a simple sort on 20 products !???? Sounds stupid no, ? well, that’s
because Javascript doesn’t HAVE a sortby 
.so 
)
It’s better for those imperative languages to HAVE a sequence
comprehension then NOT to have it. It makes the developers
in those languages (who otherwise wouldn’t have changed anyway
) more
productive
.
Now, the fact that 90% of developers like comfort and/or are learning
impaired and cannot learn something new
does NOT mean that ALL developers are lazy and/or learning impaired and
cannot learn something new
..

True, but you will still have sociological problems, with many of the old
guard protecting their turf and pitching up to code reviews and the like
objecting to the "new" code because it's "hard to understand" or it's
"non-performant".

For those who CAN learn something new, or for the fresh new students with

Post by Pavel Minaev
blank, fresh new minds
..well, teach them something better
and more productive.

What students? Students don't typically enter university these days as
virgin programmers (and haven't for a long time). They pitch up believing
they know how to program and believing they know what programming languages
they should be taught in. So professors have to run the gauntlet of why are
you teaching me this weird language instead of Java etc and how is this
going to help me get a job. I know the right answers to these questions but
universities are under pressure from students in this regard.

Post by Pavel Minaev
And that would be (according to me..) the OTHER way around: a language 95%
declarative + 5% imperative.

Erik Mejier says this is not enough.

https://queue.acm.org/detail.cfm?id=2611829

[daniela florescu]

Post by Ihe Onwuka
Erik Mejier says this is not enough.
https://queue.acm.org/detail.cfm?id=2611829 <https://queue.acm.org/detail.cfm?id=2611829>

Erik Meijer, while being an extremely smart person, is not God :-)

And I don’t see anything in this article that proves that one cannot mix those: all the
arguments he brings are for me just a reminder that, indeed, if you DO mix functional
and imperative constructs or side-effects, you better know what you are doing at compilation
and optimization time
..

That’s all.

Some details about the stuff one has to be careful about in Module 12 here — but keep in mind that
the slides are 9 years old
.:(
http://web.stanford.edu/class/cs345b/ <http://web.stanford.edu/class/cs345b/>


Best regards
Dana

[Ihe Onwuka]

Post by Ihe Onwuka
Erik Mejier says this is not enough.
https://queue.acm.org/detail.cfm?id=2611829
Erik Meijer, while being an extremely smart person, is not God :-)
And I don’t see anything in this article that proves that one cannot mix
those: all the
arguments he brings are for me just a reminder that, indeed, if you DO mix functional
and imperative constructs or side-effects, you better know what you are
doing at compilation
and optimization time
..

I'm not pretending to understand much of what he was going on about but the
headline of his article is right for an entirely different reason.

Let's step back a moment and consider what actually happens wen an
imperative language adds functional constructs - and I am not sure whether
these are intended consequences or not.

1. It puts a break on agitation to migrate to a proper functional
language. Both sides are sort of happy. The imperative people will say no
need to go to Scala/Clojure, you can do FP in Java 8. No need to go to F#
you can do FP in C# and the functional people are allowed the illusion that
they will be able to write functional code in these languages. They won't
really because the coding standards and need to conform will always be
biased towards the imperative perspective but they allow themselves to
think that they will and they do get small wins here and there.

2. Basically people carry on coding as they always have. It is not
sufficient to introduce functional constructs, to get people to use them
you have to rewire the way they think and the way they design programs.
People who do not know what mathematical induction is will not adapt to a
programming style and language constructs that requires them to think
inductively. If they have been bred to believe that recursion is
inefficient (because of languages that don't optimise tail-calls) they will
not adapt to a style of programming where that is the primary means of
effecting iteration.

So ultimately not only. do you not get anywhere near half your 95% target
(and I bet that also applies to Scala) you also stifle progress towards
migrating to a functional programming environment. Rather what you end up
with can be summarised from the Google Python coding standards. A
smattering of map and filters, some trivial list comprehensions otherwise
it's business as usual.

[daniela florescu]

I'm curious as to what exactly you believe to be features that are necessary to semi-structured data that are unique to XQuery. I've looked at the posts that you've linked to, and e.g. conditionals/typeswitches, functions and recursive functions, and try/catch, are there in pretty much any imperative language with sequence comprehensions,

Pavel,

you ignored one of the FIRST things I put there for a data processing language: the necessity to be FUNCTIONAL in nature.
(aka, 90%+ functional without side effects)

The characteristic of being “functional” is a NECESSITY for being able to do data flow analysis, aka the ability for the compiler to
know automatically what operations will be applied on each piece of data, and what data is being given as input to each operation.

Data flow analysis is a NECESSITY for optimization.

Optimization is a NECESSITY for data processing for large volumes of data.

=====

C# with select-from-where is nearly not as optimizable as XQuery with a scripting extension (as Zorba used to have it for example).

After 5 assignments you lost track of where the data came from and where the data goes, and which data changed where,
and what indexes need automatic updates.

You lost optimizability and logical/physical data independence.

That’s all.

It’s a trade-off.

Best regards
Dana



_______________________________________________
***@x-query.com
http://x-query.c

[Pavel Minaev]

I see what you're aiming at. But, in general, you can take an imperative
language and produce a declarative subset of such a language. The obvious
benefits there is that you don't have a syntactic or semantic mismatch
between your declarative islands and the imperative sea of code around
them. Declaring a function, for example, would be the same, it's just that
if you want it to be pure, then you have to avoid leaking state changes or
referencing any non-pure functions (possibly with some keyword to
explicitly enforce these restrictions).

So, for example, imagine that I take Roslyn
<https://github.com/dotnet/roslyn/wiki/Roslyn%20Overview#working-with-syntax>,
and use it to dynamically translate C# methods attributed with [Query] to
equivalent XQuery code, transparently to the user. Obviously, they'd be
limited with respect to the features of the language they are able to use
in such methods, but it's much easier to explain the limitations than it is
to explain the whole new syntax (and semantics, in some cases). I'm pretty
sure that I could have a very reasonable 1:1 mapping from such a subset to
the entirety of XQuery, actually, though some of it would probably not be
quite as pretty.

Also, correct me if I'm wrong, but it would seem that imperative patterns
(variable assignments etc) can be optimized perfectly well so long as
they're constrained (e.g. if the variable is local and no aliases are ever
leaked). Basically, so long as you can clearly contain state changes, you
can rewrite the whole thing as a purely functional monad, and that can be
analyzed and optimized same as any other declarative data flow.

Post by Pavel Minaev

Post by Pavel Minaev
I'm curious as to what exactly you believe to be features that are

necessary to semi-structured data that are unique to XQuery. I've looked at
the posts that you've linked to, and e.g. conditionals/typeswitches,
functions and recursive functions, and try/catch, are there in pretty much
any imperative language with sequence comprehensions,
Pavel,
you ignored one of the FIRST things I put there for a data processing
language: the necessity to be FUNCTIONAL in nature.
(aka, 90%+ functional without side effects)
The characteristic of being “functional” is a NECESSITY for being able to
do data flow analysis, aka the ability for the compiler to
know automatically what operations will be applied on each piece of data,
and what data is being given as input to each operation.
Data flow analysis is a NECESSITY for optimization.
Optimization is a NECESSITY for data processing for large volumes of data.
=====
C# with select-from-where is nearly not as optimizable as XQuery with a
scripting extension (as Zorba used to have it for example).
After 5 assignments you lost track of where the data came from and where
the data goes, and which data changed where,
and what indexes need automatic updates.
You lost optimizability and logical/physical data independence.
That’s all.
It’s a trade-off.
Best regards
Dana

[daniela florescu]

Pavel,

Also, correct me if I'm wrong, but it would seem that imperative patterns (variable assignments etc) can be optimized perfectly well so long as they're constrained (e.g. if the variable is local and no aliases are ever leaked). Basically, so long as you can clearly contain state changes, you can rewrite the whole thing as a purely functional monad, and that can be analyzed and optimized same as any other declarative data flow.

Huh !??? :-)

You say your developers are not productive if they use a new language, but they’ll be OK with you limiting
their beloved language that they used for millennia !?? :-)))

I think it is theoretically possible, but psychologically not feasible.

Once you give people some freedom to do something, it’s very hard to take it back. (and what about all
the legacy code!?)

Those “limits” have to be imposed directly in the language if you want faster improvements in the right direction….
otherwise I’m going to retire before this happens…:-)

Best regards
Dana







_______________________________________________
***@x-query.com
http://x-query.com/mailman/listin

[Pavel Minaev]

The beauty of a subset is that it doesn't have to be applied universally. A
single method that is responsible for querying the data (and its
dependencies), for example, can be written in such a subset, while the rest
of it remains the same (which also covers the legacy code angle).
Gradually, as isolated pieces of code are rewritten by new or more educated
developers with the new skills, they will also migrate. Obviously, if you
were to write it declaratively from the get go, you might have a very
different arrangement of pieces and interfaces between them in the first
place, so the result is less than perfect... but I'll take that over
writing for-loops for eternity. ~

And yes, it is easier. A C# developer doesn't need to be taught any new
syntax to use the subset - they only need to be broadly told what things to
avoid, and the compiler can trivially enforce it. With a new language, they
would need to start with a completely new syntax, and live with the
heretofore familiar concepts being utilized in utterly new ways (the vastly
different meaning of "return" is a very good example, actually).

Post by daniela florescu
Pavel,

Post by Pavel Minaev
Also, correct me if I'm wrong, but it would seem that imperative

patterns (variable assignments etc) can be optimized perfectly well so long
as they're constrained (e.g. if the variable is local and no aliases are
ever leaked). Basically, so long as you can clearly contain state changes,
you can rewrite the whole thing as a purely functional monad, and that can
be analyzed and optimized same as any other declarative data flow.
Huh !??? :-)
You say your developers are not productive if they use a new language, but
they’ll be OK with you limiting
their beloved language that they used for millennia !?? :-)))
I think it is theoretically possible, but psychologically not feasible.
Once you give people some freedom to do something, it’s very hard to take
it back. (and what about all
the legacy code!?)
Those “limits” have to be imposed directly in the language if you want
faster improvements in the right direction
.
otherwise I’m going to retire before this happens
:-)
Best regards
Dana

[Pavel Minaev]

C# has optional dynamic typing
<https://msdn.microsoft.com/en-us/library/dd264741.aspx>, actually. As
you'd expect, it is very natural to use with JSON
<http://www.newtonsoft.com/json/help/html/QueryJsonDynamic.htm> and similar
unstructured data.

I find this to be a good compromise, because most of the time I'm working
with data that has at least a partial schema. So I want types to be
enforced where possible, but when I need the laxity of dynamic typing, the
door is right there - and once it's opened, everything on the other side is
wonderfully dynamic without the need of any further reminders to that
effect.

As I recall, F# also has something similar, though I'm not familiar enough
with it to comment. I believe there were some considerations of adding a
similar feature to Java, as well?

Post by Pavel Minaev
The beauty of a subset is that it doesn't have to be applied universally.
A single method that is responsible for querying the data (and its
dependencies), for example, can be written in such a subset, while the rest
of it remains the same (which also covers the legacy code angle).
Gradually, as isolated pieces of code are rewritten by new or more educated
developers with the new skills, they will also migrate. Obviously, if you
were to write it declaratively from the get go, you might have a very
different arrangement of pieces and interfaces between them in the first
place, so the result is less than perfect... but I'll take that over
writing for-loops for eternity. ~
And yes, it is easier. A C# developer doesn't need to be taught any new
syntax to use the subset - they only need to be broadly told what things to
avoid, and the compiler can trivially enforce it. With a new language, they
would need to start with a completely new syntax, and live with the
heretofore familiar concepts being utilized in utterly new ways (the vastly
different meaning of "return" is a very good example, actually).

Post by daniela florescu
Pavel,

Post by Pavel Minaev
Also, correct me if I'm wrong, but it would seem that imperative

patterns (variable assignments etc) can be optimized perfectly well so long
as they're constrained (e.g. if the variable is local and no aliases are
ever leaked). Basically, so long as you can clearly contain state changes,
you can rewrite the whole thing as a purely functional monad, and that can
be analyzed and optimized same as any other declarative data flow.
Huh !??? :-)
You say your developers are not productive if they use a new language,
but they’ll be OK with you limiting
their beloved language that they used for millennia !?? :-)))
I think it is theoretically possible, but psychologically not feasible.
Once you give people some freedom to do something, it’s very hard to take
it back. (and what about all
the legacy code!?)
Those “limits” have to be imposed directly in the language if you want
faster improvements in the right direction
.
otherwise I’m going to retire before this happens
:-)
Best regards
Dana

[daniela florescu]

Pavel,

I have to investigate further, but I’ve never heard that C# can treat JSON items as native C# objects.

I don’t recall anywhere a description of what C# does when it has to grouby and the set of values are all over the map.

But let me read more, and I’ll reply to you when i am more informed.

As for schemas, that’s the entire point of semi-structured data.

Sometimes you have schemas, sometimes not.

Best regards
Dana

P.S. Even if C# would be “perfect”. It’s Microsoft. End of story. Business has other rules then technology.

C# has optional dynamic typing <https://msdn.microsoft.com/en-us/library/dd264741.aspx>, actually. As you'd expect, it is very natural to use with JSON <http://www.newtonsoft.com/json/help/html/QueryJsonDynamic.htm> and similar unstructured data.
I find this to be a good compromise, because most of the time I'm working with data that has at least a partial schema. So I want types to be enforced where possible, but when I need the laxity of dynamic typing, the door is right there - and once it's opened, everything on the other side is wonderfully dynamic without the need of any further reminders to that effect.
As I recall, F# also has something similar, though I'm not familiar enough with it to comment. I believe there were some considerations of adding a similar feature to Java, as well?
The beauty of a subset is that it doesn't have to be applied universally. A single method that is responsible for querying the data (and its dependencies), for example, can be written in such a subset, while the rest of it remains the same (which also covers the legacy code angle). Gradually, as isolated pieces of code are rewritten by new or more educated developers with the new skills, they will also migrate. Obviously, if you were to write it declaratively from the get go, you might have a very different arrangement of pieces and interfaces between them in the first place, so the result is less than perfect... but I'll take that over writing for-loops for eternity. ~
And yes, it is easier. A C# developer doesn't need to be taught any new syntax to use the subset - they only need to be broadly told what things to avoid, and the compiler can trivially enforce it. With a new language, they would need to start with a completely new syntax, and live with the heretofore familiar concepts being utilized in utterly new ways (the vastly different meaning of "return" is a very good example, actually).
Pavel,

Also, correct me if I'm wrong, but it would seem that imperative patterns (variable assignments etc) can be optimized perfectly well so long as they're constrained (e.g. if the variable is local and no aliases are ever leaked). Basically, so long as you can clearly contain state changes, you can rewrite the whole thing as a purely functional monad, and that can be analyzed and optimized same as any other declarative data flow.

Huh !??? :-)
You say your developers are not productive if they use a new language, but they’ll be OK with you limiting
their beloved language that they used for millennia !?? :-)))
I think it is theoretically possible, but psychologically not feasible.
Once you give people some freedom to do something, it’s very hard to take it back. (and what about all
the legacy code!?)
Those “limits” have to be imposed directly in the language if you want faster improvements in the right direction
.
otherwise I’m going to retire before this happens
:-)
Best regards
Dana

[Pavel Minaev]

C# by itself doesn't have any optimizations for groupby (or any other
sequence operations, for that matter). They're all just syntactic sugar for
method calls. But because those methods can capture any lambdas passed to
them as expression trees
<https://msdn.microsoft.com/en-us/library/bb397951.aspx> instead of opaque
functions, in the end you can actually get a single tree of all operations
that constitute a query, including all the invoked functions etc with a
little effort. "dynamic" is largely orthogonal to all this, except that it
basically disables the type checks to shut up the compiler and let you use
arbitrary operands for the operators - the exact semantics of your entire
query is ultimately controlled by whatever will be interpreting the
expression tree, not by the language spec.

In any case, I'm not arguing that C# is perfect in that (or any other)
respect. It merely serves as a useful example, that I am personally very
familiar with, of how far a Java-like language can be pushed if there's
desire and demand, and how an existing community of users of such a
language can eagerly adopt all those new things if they are presented in a
familiar coating. There may well be other examples, probably better ones
(Scala?), though probably none quite so popular - but I'm just using this
one because I can give you references to various features that I bring up
in this discussion off the top of my head.

(I'm also not sure why it being MSFT matters all that much these days - the
spec is published, and the reference implementation for both the compiler
and the underlying platform is now fully portable and open source under a
sane license. It's basically in the same boat as any other
non-committee-owned but otherwise open language now.)

Post by daniela florescu
Pavel,
I have to investigate further, but I’ve never heard that C# can treat JSON
items as native C# objects.
I don’t recall anywhere a description of what C# does when it has to
grouby and the set of values are all over the map.
But let me read more, and I’ll reply to you when i am more informed.
As for schemas, that’s the entire point of semi-structured data.
Sometimes you have schemas, sometimes not.
Best regards
Dana
P.S. Even if C# would be “perfect”. It’s Microsoft. End of story. Business
has other rules then technology.
C# has optional dynamic typing
<https://msdn.microsoft.com/en-us/library/dd264741.aspx>, actually. As
you'd expect, it is very natural to use with JSON
<http://www.newtonsoft.com/json/help/html/QueryJsonDynamic.htm> and
similar unstructured data.
I find this to be a good compromise, because most of the time I'm working
with data that has at least a partial schema. So I want types to be
enforced where possible, but when I need the laxity of dynamic typing, the
door is right there - and once it's opened, everything on the other side is
wonderfully dynamic without the need of any further reminders to that
effect.
As I recall, F# also has something similar, though I'm not familiar enough
with it to comment. I believe there were some considerations of adding a
similar feature to Java, as well?

Post by Pavel Minaev
The beauty of a subset is that it doesn't have to be applied universally.
A single method that is responsible for querying the data (and its
dependencies), for example, can be written in such a subset, while the rest
of it remains the same (which also covers the legacy code angle).
Gradually, as isolated pieces of code are rewritten by new or more educated
developers with the new skills, they will also migrate. Obviously, if you
were to write it declaratively from the get go, you might have a very
different arrangement of pieces and interfaces between them in the first
place, so the result is less than perfect... but I'll take that over
writing for-loops for eternity. ~
And yes, it is easier. A C# developer doesn't need to be taught any new
syntax to use the subset - they only need to be broadly told what things to
avoid, and the compiler can trivially enforce it. With a new language, they
would need to start with a completely new syntax, and live with the
heretofore familiar concepts being utilized in utterly new ways (the vastly
different meaning of "return" is a very good example, actually).

Post by daniela florescu
Pavel,

Post by Pavel Minaev
Also, correct me if I'm wrong, but it would seem that imperative

patterns (variable assignments etc) can be optimized perfectly well so long
as they're constrained (e.g. if the variable is local and no aliases are
ever leaked). Basically, so long as you can clearly contain state changes,
you can rewrite the whole thing as a purely functional monad, and that can
be analyzed and optimized same as any other declarative data flow.
Huh !??? :-)
You say your developers are not productive if they use a new language,
but they’ll be OK with you limiting
their beloved language that they used for millennia !?? :-)))
I think it is theoretically possible, but psychologically not feasible.
Once you give people some freedom to do something, it’s very hard to
take it back. (and what about all
the legacy code!?)
Those “limits” have to be imposed directly in the language if you want
faster improvements in the right direction
.
otherwise I’m going to retire before this happens
:-)
Best regards
Dana

[daniela florescu]

I see what you're aiming at. But, in general, you can take an imperative language and produce a declarative subset of such a language. The obvious benefits there is that you don't have a syntactic or semantic mismatch between your declarative islands and the imperative sea of code around them.

Pavel,

There is something ELSE that you missed in my list of requirements for processing semi-structured data.

Even if you do what you say, and you extend existing programming languages (C++, Java, C#)
.. Those are ALL 
. strongly
typed. (The only exception is Javascript, which has kind of a hacky, put together in a weekend kind of semantics
)

This strongly typed thing hardly works for semi-structured data.

Or, no, let’s be clear: ****it doesn’t work AT ALL****.

You missed another bullet in my list (might look insignificant at first sight)
. but it is ESSENTIAL if you want to deal
with semi-structured data.

4. A battery of implicit type conversions.
http://x-query.com/pipermail/talk/2015-May/004719.html <http://x-query.com/pipermail/talk/2015-May/004719.html>

Those are definitely NOT easy to define for a data processing language, I tried to explain there why.

Again, among all the programming languages that I know, XQuery has the best (ONLY!?) set of such operations.

And again, those operations have NOTHING to do with XML. They are completely orthogonal to XML.

Best regards
Dana

[daniela florescu]

I think that was distinguishes LINQ is not that it was the first such experiment, but that it was the first one that happened in a mainstream non-declarative language (as opposed to a fork of one) and was widely adopted by the community of that language.

Pavel,


Yep. You are right. For that give credit to Bill Gates, who allowed this “new” and potentially disruptive thing to happen
in a large scale industrial product.

This was not a “first” as research, but was definitely a “first” as business.

Bill Gates was not only a brilliant businessman, but ALSO loved and understood technology.
(too bad he had no aesthetic and usability sense though :-))

======

Note to the CEO of MarkLogic
. can I tell you something, dear CEO !? THAT’s the difference between you (on one side)
and Larry Ellison and Bill Gates and Steve Jobs (on the other..).

The courage to try something completely new and untested, and STICK to what you believe in.

Not jump from XQuery on server side to Javascript on server side because the wind blows sideways 
 or someone influenced you
in the corridor 

.like you did.

Best regards
Dana

P.S. And if you ask me why the business mistakes of MarkLogic are MY business
.it’s very simple.

If MarkLogic screws up financially
 it will drag down THE WHOLE XQUERY TECHNOLOGY.

And I spent 20 years working on it

.so I take it personally.